2024-05-17T14:40:42.485 INFO:root:teuthology version: 0.0.1.dev276+gcad7edd 2024-05-17T14:40:42.486 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_dis@teuthology --archive /home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219 --name dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi --verbose --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} -- /home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219/orig.config.yaml 2024-05-17T14:40:42.531 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-17T14:40:42.612 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219 branch: wip-dis-testing description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} email: idryomov@gmail.com first_in_suite: false job_id: '7711219' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-dis-testing ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd memory target: 1610612736 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: ee59d86aad04089481e55c1a15aa5871339935b4 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: ee59d86aad04089481e55c1a15aa5871339935b4 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing sha1: ee59d86aad04089481e55c1a15aa5871339935b4 owner: scheduled_dis@teuthology priority: 70 repo: https://github.com/ceph/ceph-ci roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 3754 sha1: ee59d86aad04089481e55c1a15aa5871339935b4 sleep_before_teardown: 0 subset: 56/99 suite: rbd suite_branch: wip-dis-testing suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci suite_sha1: ee59d86aad04089481e55c1a15aa5871339935b4 targets: smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHYe1ZGnxZjX7+sQweRvnThF1q47PodBSwam7dcyTHgN8O9FeALEpBU/YCmwo/S2di3fktueeGSo22ojmFO2hnY= smithi195.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHOwvwP+xWhFfbARMc1jjL85g4dIJywrcGaxZwuxpRdYo+R7TPpbE+bquiE8iV2Wra/zwrM6BG46opM+s86KYvw= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: wip-fix-extra-system-packages teuthology_sha1: cad7edd8d899a7c59a48f7da3294fca4cb778f64 timestamp: 2024-05-17_14:29:27 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-17T14:40:42.612 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa; will attempt to use it 2024-05-17T14:40:42.613 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks 2024-05-17T14:40:42.613 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-17T14:40:42.614 INFO:teuthology.task.internal:Checking packages... 2024-05-17T14:40:43.026 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ee59d86aad04089481e55c1a15aa5871339935b4' 2024-05-17T14:40:43.026 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-17T14:40:43.026 INFO:teuthology.packaging:ref: None 2024-05-17T14:40:43.026 INFO:teuthology.packaging:tag: None 2024-05-17T14:40:43.026 INFO:teuthology.packaging:branch: wip-dis-testing 2024-05-17T14:40:43.026 INFO:teuthology.packaging:sha1: ee59d86aad04089481e55c1a15aa5871339935b4 2024-05-17T14:40:43.026 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-dis-testing 2024-05-17T14:40:43.169 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3735.gee59d86a 2024-05-17T14:40:43.171 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-17T14:40:43.177 INFO:teuthology.task.internal:no buildpackages task found 2024-05-17T14:40:43.177 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-17T14:40:43.191 INFO:teuthology.task.internal:Saving configuration 2024-05-17T14:40:43.208 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-17T14:40:43.215 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-17T14:40:43.239 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-17 14:35:49.117998', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHYe1ZGnxZjX7+sQweRvnThF1q47PodBSwam7dcyTHgN8O9FeALEpBU/YCmwo/S2di3fktueeGSo22ojmFO2hnY='} 2024-05-17T14:40:43.259 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-17 14:35:49.119830', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHOwvwP+xWhFfbARMc1jjL85g4dIJywrcGaxZwuxpRdYo+R7TPpbE+bquiE8iV2Wra/zwrM6BG46opM+s86KYvw='} 2024-05-17T14:40:43.260 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-17T14:40:43.268 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-05-17T14:40:43.268 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-05-17T14:40:43.268 INFO:teuthology.run_tasks:Running task console_log... 2024-05-17T14:40:43.474 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4ede64fac0>, signals=[15]) 2024-05-17T14:40:43.474 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-17T14:40:43.483 INFO:teuthology.task.internal:Opening connections... 2024-05-17T14:40:43.483 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-05-17T14:40:43.484 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-17T14:40:43.559 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2024-05-17T14:40:43.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-17T14:40:43.633 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-17T14:40:43.640 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-05-17T14:40:43.657 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-05-17T14:40:43.657 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-05-17T14:40:43.715 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2024-05-17T14:40:43.715 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="9" 2024-05-17T14:40:43.715 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2024-05-17T14:40:43.715 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2024-05-17T14:40:43.715 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="9" 2024-05-17T14:40:43.715 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el9" 2024-05-17T14:40:43.715 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-17T14:40:43.715 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2024-05-17T14:40:43.715 INFO:teuthology.orchestra.run.smithi134.stdout:LOGO="fedora-logo-icon" 2024-05-17T14:40:43.715 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-17T14:40:43.715 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2024-05-17T14:40:43.715 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-17T14:40:43.715 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-17T14:40:43.716 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-17T14:40:43.716 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-05-17T14:40:43.742 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2024-05-17T14:40:43.759 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2024-05-17T14:40:43.760 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2024-05-17T14:40:43.817 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream" 2024-05-17T14:40:43.817 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="9" 2024-05-17T14:40:43.817 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2024-05-17T14:40:43.818 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2024-05-17T14:40:43.818 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="9" 2024-05-17T14:40:43.818 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el9" 2024-05-17T14:40:43.818 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-17T14:40:43.818 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2024-05-17T14:40:43.818 INFO:teuthology.orchestra.run.smithi195.stdout:LOGO="fedora-logo-icon" 2024-05-17T14:40:43.818 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-17T14:40:43.818 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2024-05-17T14:40:43.818 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-17T14:40:43.818 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-17T14:40:43.818 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-17T14:40:43.819 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2024-05-17T14:40:43.842 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-17T14:40:43.853 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-17T14:40:43.867 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-17T14:40:43.867 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-05-17T14:40:43.869 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2024-05-17T14:40:43.886 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-17T14:40:43.960 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-17T14:40:43.960 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-05-17T14:40:43.963 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2024-05-17T14:40:43.994 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-17T14:40:44.061 INFO:teuthology.run_tasks:Running task kernel... 2024-05-17T14:40:44.114 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-05-17T14:40:44.114 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-05-17T14:40:44.115 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-17T14:40:44.115 DEBUG:teuthology.orchestra.run.smithi195:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-17T14:40:44.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-17T14:40:44.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-17T14:40:44.133 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-05-17T14:40:44.133 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-05-17T14:40:44.191 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-437.el9.x86_64 2024-05-17T14:40:44.191 INFO:teuthology.orchestra.run.smithi195.stdout:5.14.0-437.el9.x86_64 2024-05-17T14:40:44.191 INFO:teuthology.task.kernel:Running kernel on smithi134: 5.14.0-437.el9.x86_64 2024-05-17T14:40:44.191 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum install -y kernel 2024-05-17T14:40:44.191 INFO:teuthology.task.kernel:Running kernel on smithi195: 5.14.0-437.el9.x86_64 2024-05-17T14:40:44.191 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum install -y kernel 2024-05-17T14:40:46.410 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.1 MB 00:00 2024-05-17T14:40:46.677 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.1 MB 00:00 2024-05-17T14:40:48.186 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-05-17T14:40:53.228 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.2 MB 00:01 2024-05-17T14:40:55.761 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 16 kB 00:01 2024-05-17T14:40:57.230 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-05-17T14:41:02.363 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-05-17T14:41:03.680 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-05-17T14:41:04.311 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-17T14:41:04.311 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-17T14:41:04.397 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:41:04.404 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:41:04.404 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-05-17T14:41:04.404 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:41:04.404 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-05-17T14:41:04.404 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 5.14.0-446.el9 baseos 550 k 2024-05-17T14:41:04.405 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-05-17T14:41:04.405 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core x86_64 5.14.0-446.el9 baseos 15 M 2024-05-17T14:41:04.405 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules x86_64 5.14.0-446.el9 baseos 34 M 2024-05-17T14:41:04.405 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core x86_64 5.14.0-446.el9 baseos 28 M 2024-05-17T14:41:04.405 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:41:04.405 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-05-17T14:41:04.405 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:41:04.405 INFO:teuthology.orchestra.run.smithi134.stdout:Install 4 Packages 2024-05-17T14:41:04.405 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:41:04.406 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 78 M 2024-05-17T14:41:04.406 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 123 M 2024-05-17T14:41:04.406 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-05-17T14:41:04.914 INFO:teuthology.orchestra.run.smithi134.stdout:(1/4): kernel-5.14.0-446.el9.x86_64.rpm 2.7 MB/s | 550 kB 00:00 2024-05-17T14:41:05.415 INFO:teuthology.orchestra.run.smithi134.stdout:(2/4): kernel-core-5.14.0-446.el9.x86_64.rpm 22 MB/s | 15 MB 00:00 2024-05-17T14:41:05.957 INFO:teuthology.orchestra.run.smithi134.stdout:(3/4): kernel-modules-5.14.0-446.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-05-17T14:41:06.415 INFO:teuthology.orchestra.run.smithi134.stdout:(4/4): kernel-modules-core-5.14.0-446.el9.x86_6 19 MB/s | 28 MB 00:01 2024-05-17T14:41:06.415 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:41:06.416 INFO:teuthology.orchestra.run.smithi134.stdout:Total 39 MB/s | 78 MB 00:02 2024-05-17T14:41:07.105 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-05-17T14:41:07.424 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-05-17T14:41:07.424 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-05-17T14:41:08.341 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-05-17T14:41:08.341 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-05-17T14:41:10.727 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-05-17T14:41:10.739 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - CRB 1.8 MB/s | 6.2 MB 00:03 2024-05-17T14:41:11.407 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-core-5.14.0-446.el9.x86_64 1/4 2024-05-17T14:41:11.498 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-17T14:41:12.615 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-05-17T14:41:14.167 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-17T14:41:14.300 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-17T14:41:16.216 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 6.1 MB/s | 21 MB 00:03 2024-05-17T14:41:18.976 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-17T14:41:19.086 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-5.14.0-446.el9.x86_64 4/4 2024-05-17T14:41:21.458 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-05-17T14:41:23.414 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-17T14:41:23.414 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-17T14:41:23.504 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:41:23.516 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:41:23.516 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-05-17T14:41:23.516 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:41:23.516 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-17T14:41:23.517 INFO:teuthology.orchestra.run.smithi195.stdout: kernel x86_64 5.14.0-446.el9 baseos 550 k 2024-05-17T14:41:23.517 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-17T14:41:23.517 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core x86_64 5.14.0-446.el9 baseos 15 M 2024-05-17T14:41:23.517 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules x86_64 5.14.0-446.el9 baseos 34 M 2024-05-17T14:41:23.517 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-core x86_64 5.14.0-446.el9 baseos 28 M 2024-05-17T14:41:23.517 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:41:23.517 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-17T14:41:23.517 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:41:23.517 INFO:teuthology.orchestra.run.smithi195.stdout:Install 4 Packages 2024-05-17T14:41:23.517 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:41:23.518 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 78 M 2024-05-17T14:41:23.518 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 123 M 2024-05-17T14:41:23.518 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-17T14:41:23.833 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-core-5.14.0-446.el9.x86_64 4/4 2024-05-17T14:41:24.409 INFO:teuthology.orchestra.run.smithi195.stdout:(1/4): kernel-5.14.0-446.el9.x86_64.rpm 713 kB/s | 550 kB 00:00 2024-05-17T14:41:27.567 INFO:teuthology.orchestra.run.smithi195.stdout:(2/4): kernel-core-5.14.0-446.el9.x86_64.rpm 3.9 MB/s | 15 MB 00:03 2024-05-17T14:41:37.083 INFO:teuthology.orchestra.run.smithi195.stdout:(3/4): kernel-modules-core-5.14.0-446.el9.x86_6 2.2 MB/s | 28 MB 00:12 2024-05-17T14:41:40.025 INFO:teuthology.orchestra.run.smithi195.stdout:(4/4): kernel-modules-5.14.0-446.el9.x86_64.rpm 2.1 MB/s | 34 MB 00:16 2024-05-17T14:41:40.025 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:41:40.025 INFO:teuthology.orchestra.run.smithi195.stdout:Total 4.7 MB/s | 78 MB 00:16 2024-05-17T14:41:40.706 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-17T14:41:41.045 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-17T14:41:41.046 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-17T14:41:41.988 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-17T14:41:41.988 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-17T14:41:44.403 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-17T14:41:45.020 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-core-5.14.0-446.el9.x86_64 1/4 2024-05-17T14:41:45.108 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-17T14:41:47.741 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-17T14:41:47.879 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-17T14:41:52.545 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-17T14:41:52.590 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-5.14.0-446.el9.x86_64 4/4 2024-05-17T14:41:53.069 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-446.el9.x86_64 4/4 2024-05-17T14:41:53.108 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-446.el9.x86_64 4/4 2024-05-17T14:41:53.516 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-5.14.0-446.el9.x86_64 4/4 2024-05-17T14:41:53.516 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-5.14.0-446.el9.x86_64 1/4 2024-05-17T14:41:53.516 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-17T14:41:53.516 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-17T14:41:54.243 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-core-5.14.0-446.el9.x86_64 4/4 2024-05-17T14:41:54.243 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:41:54.243 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-05-17T14:41:54.243 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-5.14.0-446.el9.x86_64 2024-05-17T14:41:54.244 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core-5.14.0-446.el9.x86_64 2024-05-17T14:41:54.244 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-5.14.0-446.el9.x86_64 2024-05-17T14:41:54.244 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core-5.14.0-446.el9.x86_64 2024-05-17T14:41:54.244 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:41:54.244 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:41:54.615 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-05-17T14:41:54.685 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-446.el9.x86_64 2024-05-17T14:41:54.685 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-446.el9.x86_64 2024-05-17T14:41:54.685 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-446.el9.x86_64 2024-05-17T14:41:54.686 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-17T14:41:54.686 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-17T14:41:54.686 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-05-17T14:41:54.686 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2024-05-17T14:41:54.686 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-05-17T14:41:54.736 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-446.el9.x86_64 2024-05-17T14:41:54.736 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-446.el9.x86_64 2024-05-17T14:41:54.736 DEBUG:teuthology.orchestra.run.smithi134:> sudo rpm -qi grub2-tools 2024-05-17T14:41:54.784 INFO:teuthology.orchestra.run.smithi134.stdout:Name : grub2-tools 2024-05-17T14:41:54.785 INFO:teuthology.orchestra.run.smithi134.stdout:Epoch : 1 2024-05-17T14:41:54.785 INFO:teuthology.orchestra.run.smithi134.stdout:Version : 2.06 2024-05-17T14:41:54.785 INFO:teuthology.orchestra.run.smithi134.stdout:Release : 68.el9 2024-05-17T14:41:54.785 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: x86_64 2024-05-17T14:41:54.785 INFO:teuthology.orchestra.run.smithi134.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-17T14:41:54.785 INFO:teuthology.orchestra.run.smithi134.stdout:Group : Unspecified 2024-05-17T14:41:54.785 INFO:teuthology.orchestra.run.smithi134.stdout:Size : 8264254 2024-05-17T14:41:54.785 INFO:teuthology.orchestra.run.smithi134.stdout:License : GPLv3+ 2024-05-17T14:41:54.785 INFO:teuthology.orchestra.run.smithi134.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-17T14:41:54.785 INFO:teuthology.orchestra.run.smithi134.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-17T14:41:54.785 INFO:teuthology.orchestra.run.smithi134.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-17T14:41:54.785 INFO:teuthology.orchestra.run.smithi134.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-17T14:41:54.785 INFO:teuthology.orchestra.run.smithi134.stdout:Packager : builder@centos.org 2024-05-17T14:41:54.786 INFO:teuthology.orchestra.run.smithi134.stdout:Vendor : CentOS 2024-05-17T14:41:54.786 INFO:teuthology.orchestra.run.smithi134.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-17T14:41:54.786 INFO:teuthology.orchestra.run.smithi134.stdout:Summary : Support tools for GRUB. 2024-05-17T14:41:54.786 INFO:teuthology.orchestra.run.smithi134.stdout:Description : 2024-05-17T14:41:54.786 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:41:54.786 INFO:teuthology.orchestra.run.smithi134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-17T14:41:54.786 INFO:teuthology.orchestra.run.smithi134.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-17T14:41:54.786 INFO:teuthology.orchestra.run.smithi134.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-17T14:41:54.786 INFO:teuthology.orchestra.run.smithi134.stdout:hardware devices. 2024-05-17T14:41:54.786 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:41:54.786 INFO:teuthology.orchestra.run.smithi134.stdout:This subpackage provides tools for support of all platforms. 2024-05-17T14:41:54.787 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-17T14:41:54.787 INFO:teuthology.task.kernel:Updating grub on smithi134 to boot 5.14.0-446.el9.x86_64 2024-05-17T14:41:54.787 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-17T14:41:55.360 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2024-05-17T14:41:56.790 INFO:teuthology.orchestra.run.smithi134.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-17T14:41:56.820 INFO:teuthology.orchestra.run.smithi134.stderr:done 2024-05-17T14:41:56.823 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-05-17T14:41:56.838 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.MtoEQSffzD 2024-05-17T14:41:56.838 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MtoEQSffzD 2024-05-17T14:41:56.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.MtoEQSffzD 2024-05-17T14:41:57.037 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.MtoEQSffzD is 6KB 2024-05-17T14:41:57.086 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.MtoEQSffzD 2024-05-17T14:41:57.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo /bin/ls /boot/loader/entries || true 2024-05-17T14:41:57.161 INFO:teuthology.orchestra.run.smithi134.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-17T14:41:57.161 INFO:teuthology.orchestra.run.smithi134.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-17T14:41:57.161 INFO:teuthology.orchestra.run.smithi134.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-17T14:41:57.162 INFO:teuthology.orchestra.run.smithi134.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-446.el9.x86_64.conf 2024-05-17T14:41:57.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-446.el9.x86_64 2024-05-17T14:41:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-core-5.14.0-446.el9.x86_64 4/4 2024-05-17T14:41:57.313 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2024-05-17T14:42:26.564 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-5.14.0-446.el9.x86_64 4/4 2024-05-17T14:42:26.602 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-5.14.0-446.el9.x86_64 4/4 2024-05-17T14:42:26.954 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-5.14.0-446.el9.x86_64 4/4 2024-05-17T14:42:26.954 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-5.14.0-446.el9.x86_64 1/4 2024-05-17T14:42:26.955 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-17T14:42:26.955 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-17T14:42:27.316 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-05-17T14:42:27.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-05-17T14:42:27.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-17T14:42:27.779 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-core-5.14.0-446.el9.x86_64 4/4 2024-05-17T14:42:27.780 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:42:27.780 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-17T14:42:27.780 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-5.14.0-446.el9.x86_64 2024-05-17T14:42:27.780 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core-5.14.0-446.el9.x86_64 2024-05-17T14:42:27.780 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-5.14.0-446.el9.x86_64 2024-05-17T14:42:27.780 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-core-5.14.0-446.el9.x86_64 2024-05-17T14:42:27.780 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:42:27.780 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:42:28.113 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-05-17T14:42:28.226 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-5.14.0-446.el9.x86_64 2024-05-17T14:42:28.226 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-446.el9.x86_64 2024-05-17T14:42:28.227 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-446.el9.x86_64 2024-05-17T14:42:28.227 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-17T14:42:28.227 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-17T14:42:28.227 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-05-17T14:42:28.227 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi195.front.sepia.ceph.com, path=None, version=distro) 2024-05-17T14:42:28.227 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-05-17T14:42:28.277 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-5.14.0-446.el9.x86_64 2024-05-17T14:42:28.277 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-446.el9.x86_64 2024-05-17T14:42:28.277 DEBUG:teuthology.orchestra.run.smithi195:> sudo rpm -qi grub2-tools 2024-05-17T14:42:28.307 INFO:teuthology.orchestra.run.smithi195.stdout:Name : grub2-tools 2024-05-17T14:42:28.308 INFO:teuthology.orchestra.run.smithi195.stdout:Epoch : 1 2024-05-17T14:42:28.308 INFO:teuthology.orchestra.run.smithi195.stdout:Version : 2.06 2024-05-17T14:42:28.308 INFO:teuthology.orchestra.run.smithi195.stdout:Release : 68.el9 2024-05-17T14:42:28.308 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: x86_64 2024-05-17T14:42:28.308 INFO:teuthology.orchestra.run.smithi195.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-17T14:42:28.308 INFO:teuthology.orchestra.run.smithi195.stdout:Group : Unspecified 2024-05-17T14:42:28.308 INFO:teuthology.orchestra.run.smithi195.stdout:Size : 8264254 2024-05-17T14:42:28.308 INFO:teuthology.orchestra.run.smithi195.stdout:License : GPLv3+ 2024-05-17T14:42:28.308 INFO:teuthology.orchestra.run.smithi195.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-17T14:42:28.308 INFO:teuthology.orchestra.run.smithi195.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-17T14:42:28.308 INFO:teuthology.orchestra.run.smithi195.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-17T14:42:28.309 INFO:teuthology.orchestra.run.smithi195.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-17T14:42:28.309 INFO:teuthology.orchestra.run.smithi195.stdout:Packager : builder@centos.org 2024-05-17T14:42:28.309 INFO:teuthology.orchestra.run.smithi195.stdout:Vendor : CentOS 2024-05-17T14:42:28.309 INFO:teuthology.orchestra.run.smithi195.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-17T14:42:28.309 INFO:teuthology.orchestra.run.smithi195.stdout:Summary : Support tools for GRUB. 2024-05-17T14:42:28.309 INFO:teuthology.orchestra.run.smithi195.stdout:Description : 2024-05-17T14:42:28.309 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:42:28.309 INFO:teuthology.orchestra.run.smithi195.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-17T14:42:28.309 INFO:teuthology.orchestra.run.smithi195.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-17T14:42:28.309 INFO:teuthology.orchestra.run.smithi195.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-17T14:42:28.309 INFO:teuthology.orchestra.run.smithi195.stdout:hardware devices. 2024-05-17T14:42:28.309 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:42:28.309 INFO:teuthology.orchestra.run.smithi195.stdout:This subpackage provides tools for support of all platforms. 2024-05-17T14:42:28.310 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-17T14:42:28.310 INFO:teuthology.task.kernel:Updating grub on smithi195 to boot 5.14.0-446.el9.x86_64 2024-05-17T14:42:28.310 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-17T14:42:28.988 INFO:teuthology.orchestra.run.smithi195.stderr:Generating grub configuration file ... 2024-05-17T14:42:30.384 INFO:teuthology.orchestra.run.smithi195.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-17T14:42:30.421 INFO:teuthology.orchestra.run.smithi195.stderr:done 2024-05-17T14:42:30.423 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2024-05-17T14:42:30.439 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.wlM4mu3Ahl 2024-05-17T14:42:30.440 DEBUG:teuthology.orchestra.run.smithi195:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wlM4mu3Ahl 2024-05-17T14:42:30.514 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.wlM4mu3Ahl 2024-05-17T14:42:30.651 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.wlM4mu3Ahl is 6KB 2024-05-17T14:42:30.700 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.wlM4mu3Ahl 2024-05-17T14:42:30.714 DEBUG:teuthology.orchestra.run.smithi195:> sudo /bin/ls /boot/loader/entries || true 2024-05-17T14:42:30.776 INFO:teuthology.orchestra.run.smithi195.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-17T14:42:30.776 INFO:teuthology.orchestra.run.smithi195.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-17T14:42:30.776 INFO:teuthology.orchestra.run.smithi195.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-17T14:42:30.776 INFO:teuthology.orchestra.run.smithi195.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-446.el9.x86_64.conf 2024-05-17T14:42:30.777 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-446.el9.x86_64 2024-05-17T14:42:30.918 DEBUG:teuthology.orchestra.run.smithi195:> sudo shutdown -r now 2024-05-17T14:43:00.950 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-05-17T14:43:00.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-05-17T14:43:00.953 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-17T14:43:19.455 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-05-17T14:43:27.325 DEBUG:teuthology.orchestra.remote:timed out 2024-05-17T14:43:28.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-05-17T14:43:28.458 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-17T14:43:36.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-05-17T14:43:36.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-17T14:43:36.735 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-05-17T14:43:37.240 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-05-17T14:43:37.240 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-446.el9.x86_64"... 2024-05-17T14:43:37.240 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-05-17T14:43:37.254 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-446.el9.x86_64 2024-05-17T14:43:37.254 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-446.el9.x86_64 vs 5.14.0-446.el9.x86_64 2024-05-17T14:43:37.254 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-17T14:43:37.255 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-17T14:43:38.255 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-05-17T14:43:38.256 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-17T14:43:38.335 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2024-05-17T14:43:38.370 DEBUG:teuthology.parallel:result is None 2024-05-17T14:44:28.460 DEBUG:teuthology.orchestra.remote:timed out 2024-05-17T14:44:40.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-05-17T14:44:40.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-17T14:44:40.779 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-05-17T14:44:41.147 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-05-17T14:44:41.147 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-446.el9.x86_64"... 2024-05-17T14:44:41.148 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-05-17T14:44:41.164 INFO:teuthology.orchestra.run.smithi195.stdout:5.14.0-446.el9.x86_64 2024-05-17T14:44:41.164 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-446.el9.x86_64 vs 5.14.0-446.el9.x86_64 2024-05-17T14:44:41.164 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-17T14:44:41.164 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-17T14:44:42.165 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-05-17T14:44:42.165 DEBUG:teuthology.orchestra.run.smithi195:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-17T14:44:42.233 INFO:teuthology.orchestra.run.smithi195.stdout:ttyS1 2024-05-17T14:44:42.271 DEBUG:teuthology.parallel:result is None 2024-05-17T14:44:42.271 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-17T14:44:42.279 INFO:teuthology.task.internal:Creating test directory... 2024-05-17T14:44:42.279 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-17T14:44:42.283 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-17T14:44:42.301 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-17T14:44:42.310 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-17T14:44:42.322 INFO:teuthology.task.internal:Creating archive directory... 2024-05-17T14:44:42.323 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-17T14:44:42.343 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-17T14:44:42.392 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-17T14:44:42.401 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-17T14:44:42.401 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-17T14:44:42.404 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-17T14:44:42.441 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-17T14:44:42.452 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-17T14:44:42.457 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-17T14:44:42.467 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-17T14:44:42.469 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-17T14:44:42.480 INFO:teuthology.task.internal:Configuring sudo... 2024-05-17T14:44:42.480 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-17T14:44:42.501 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-17T14:44:42.536 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-17T14:44:42.546 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-17T14:44:42.547 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-17T14:44:42.568 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-17T14:44:42.592 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-17T14:44:42.667 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-17T14:44:42.726 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-17T14:44:42.767 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-17T14:44:42.810 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:44:42.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-17T14:44:42.875 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-17T14:44:42.913 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-17T14:44:42.976 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-17T14:44:43.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-17T14:44:43.106 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:44:43.106 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-17T14:44:43.174 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-05-17T14:44:43.176 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2024-05-17T14:44:43.226 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-17T14:44:43.269 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-17T14:44:43.729 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-17T14:44:43.738 INFO:teuthology.task.internal:Starting timer... 2024-05-17T14:44:43.738 INFO:teuthology.run_tasks:Running task pcp... 2024-05-17T14:44:43.753 INFO:teuthology.run_tasks:Running task selinux... 2024-05-17T14:44:43.766 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-05-17T14:44:43.766 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2024-05-17T14:44:43.840 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2024-05-17T14:44:43.842 DEBUG:teuthology.orchestra.run.smithi195:> sudo service auditd rotate 2024-05-17T14:44:43.918 INFO:teuthology.orchestra.run.smithi195.stdout:Rotating logs: 2024-05-17T14:44:43.920 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-17T14:44:43.920 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2024-05-17T14:44:43.950 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2024-05-17T14:44:43.950 DEBUG:teuthology.orchestra.run.smithi195:> /usr/sbin/getenforce 2024-05-17T14:44:44.027 INFO:teuthology.orchestra.run.smithi195.stdout:Permissive 2024-05-17T14:44:44.028 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi134.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi195.front.sepia.ceph.com': 'permissive'} 2024-05-17T14:44:44.028 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-17T14:44:44.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-17T14:44:44.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-17T14:44:44.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-17T14:44:44.083 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-17T14:44:44.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2024-05-17T14:44:44.121 DEBUG:teuthology.orchestra.run.smithi195:> sudo /usr/sbin/setenforce permissive 2024-05-17T14:44:44.150 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-17T14:44:44.159 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-17T14:44:44.170 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-17T14:44:44.171 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi134.front.sepia.ceph.com,smithi195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-17T14:49:01.055 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi134.front.sepia.ceph.com'), Remote(name='ubuntu@smithi195.front.sepia.ceph.com')] 2024-05-17T14:49:01.057 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-05-17T14:49:01.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-17T14:49:01.143 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-05-17T14:49:01.226 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-05-17T14:49:01.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-05-17T14:49:01.227 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-17T14:49:01.299 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-05-17T14:49:01.387 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-05-17T14:49:01.387 INFO:teuthology.run_tasks:Running task clock... 2024-05-17T14:49:01.422 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-17T14:49:01.422 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-17T14:49:01.423 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-17T14:49:01.425 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-17T14:49:01.426 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-17T14:49:01.456 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-17T14:49:01.458 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-17T14:49:01.471 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-17T14:49:01.473 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-17T14:49:01.497 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: ntpd: command not found 2024-05-17T14:49:01.498 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2024-05-17T14:49:01.508 INFO:teuthology.orchestra.run.smithi195.stdout:506 Cannot talk to daemon 2024-05-17T14:49:01.509 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2024-05-17T14:49:01.522 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-17T14:49:01.524 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-17T14:49:01.536 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-17T14:49:01.538 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-17T14:49:01.588 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-05-17T14:49:01.591 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-17T14:49:01.591 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-05-17T14:49:01.592 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-17T14:49:01.592 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-17T14:49:01.592 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-17T14:49:01.592 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-17T14:49:01.595 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-05-17T14:49:01.599 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-17T14:49:01.599 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-05-17T14:49:01.599 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-17T14:49:01.599 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-17T14:49:01.599 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-17T14:49:01.599 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-17T14:49:01.600 INFO:teuthology.run_tasks:Running task install... 2024-05-17T14:49:01.607 DEBUG:teuthology.task.install:project ceph 2024-05-17T14:49:01.608 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4'}} 2024-05-17T14:49:01.608 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4'} 2024-05-17T14:49:01.608 INFO:teuthology.task.install:Using flavor: default 2024-05-17T14:49:01.613 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-17T14:49:01.613 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-05-17T14:49:01.614 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4', 'tag': None, 'wait_for_package': False} 2024-05-17T14:49:01.614 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ee59d86aad04089481e55c1a15aa5871339935b4 2024-05-17T14:49:01.616 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4', 'tag': None, 'wait_for_package': False} 2024-05-17T14:49:01.616 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ee59d86aad04089481e55c1a15aa5871339935b4 2024-05-17T14:49:01.757 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-dis-testing/ee59d86aad04089481e55c1a15aa5871339935b4/centos/9/flavors/default/ 2024-05-17T14:49:01.757 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3735.gee59d86a 2024-05-17T14:49:01.837 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-dis-testing/ee59d86aad04089481e55c1a15aa5871339935b4/centos/9/flavors/default/ 2024-05-17T14:49:01.837 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3735.gee59d86a 2024-05-17T14:49:01.943 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-dis-testing/ee59d86aad04089481e55c1a15aa5871339935b4/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-dis-testing/ee59d86aad04089481e55c1a15aa5871339935b4/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-dis-testing/ee59d86aad04089481e55c1a15aa5871339935b4/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-17T14:49:01.943 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:49:01.943 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-17T14:49:01.970 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-05-17T14:49:01.970 DEBUG:teuthology.orchestra.run.smithi195:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ee59d86aad04089481e55c1a15aa5871339935b4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-17T14:49:02.038 DEBUG:teuthology.orchestra.run.smithi195:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-17T14:49:02.042 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-dis-testing/ee59d86aad04089481e55c1a15aa5871339935b4/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-dis-testing/ee59d86aad04089481e55c1a15aa5871339935b4/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-dis-testing/ee59d86aad04089481e55c1a15aa5871339935b4/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-17T14:49:02.043 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:49:02.043 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-17T14:49:02.070 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-05-17T14:49:02.070 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ee59d86aad04089481e55c1a15aa5871339935b4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-17T14:49:02.114 DEBUG:teuthology.orchestra.run.smithi195:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-17T14:49:02.136 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-17T14:49:02.179 INFO:teuthology.orchestra.run.smithi195.stdout:check_obsoletes = 1 2024-05-17T14:49:02.180 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum clean all 2024-05-17T14:49:02.212 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-17T14:49:02.276 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2024-05-17T14:49:02.278 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2024-05-17T14:49:02.413 INFO:teuthology.orchestra.run.smithi195.stdout:57 files removed 2024-05-17T14:49:02.443 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-radosgw 2024-05-17T14:49:02.528 INFO:teuthology.orchestra.run.smithi134.stdout:57 files removed 2024-05-17T14:49:02.556 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw 2024-05-17T14:49:03.248 INFO:teuthology.orchestra.run.smithi195.stdout:ceph packages for x86_64 157 kB/s | 84 kB 00:00 2024-05-17T14:49:03.525 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 120 kB/s | 84 kB 00:00 2024-05-17T14:49:03.714 INFO:teuthology.orchestra.run.smithi195.stdout:ceph noarch packages 28 kB/s | 12 kB 00:00 2024-05-17T14:49:04.000 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 28 kB/s | 12 kB 00:00 2024-05-17T14:49:04.340 INFO:teuthology.orchestra.run.smithi195.stdout:ceph source packages 3.3 kB/s | 1.9 kB 00:00 2024-05-17T14:49:04.476 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 4.4 kB/s | 1.9 kB 00:00 2024-05-17T14:49:09.592 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 8.1 MB 00:05 2024-05-17T14:49:09.715 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - BaseOS 1.5 MB/s | 8.1 MB 00:05 2024-05-17T14:49:14.024 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 19 MB 00:03 2024-05-17T14:49:16.751 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 3.1 MB/s | 19 MB 00:06 2024-05-17T14:49:17.950 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.2 MB 00:00 2024-05-17T14:49:19.442 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 16 kB 00:00 2024-05-17T14:49:21.305 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-05-17T14:49:21.610 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 6.2 MB 00:01 2024-05-17T14:49:23.177 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 16 kB 00:00 2024-05-17T14:49:24.870 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-05-17T14:49:26.502 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-05-17T14:49:28.794 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:49:28.808 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:49:28.808 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-17T14:49:28.808 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:49:28.808 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-17T14:49:28.808 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 11 M 2024-05-17T14:49:28.809 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrading: 2024-05-17T14:49:28.809 INFO:teuthology.orchestra.run.smithi195.stdout: librados2 x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 3.4 M 2024-05-17T14:49:28.809 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1 x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 3.1 M 2024-05-17T14:49:28.809 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-17T14:49:28.809 INFO:teuthology.orchestra.run.smithi195.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-17T14:49:28.809 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 5.4 M 2024-05-17T14:49:28.809 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 21 M 2024-05-17T14:49:28.809 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 25 k 2024-05-17T14:49:28.809 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-17T14:49:28.809 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-17T14:49:28.809 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-17T14:49:28.809 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2 x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 748 k 2024-05-17T14:49:28.810 INFO:teuthology.orchestra.run.smithi195.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-17T14:49:28.810 INFO:teuthology.orchestra.run.smithi195.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-17T14:49:28.810 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-17T14:49:28.810 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1 x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 501 k 2024-05-17T14:49:28.810 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-17T14:49:28.810 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2 x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 5.4 M 2024-05-17T14:49:28.810 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-17T14:49:28.810 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-17T14:49:28.810 INFO:teuthology.orchestra.run.smithi195.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-17T14:49:28.810 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 45 k 2024-05-17T14:49:28.810 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 136 k 2024-05-17T14:49:28.810 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 162 k 2024-05-17T14:49:28.810 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-17T14:49:28.811 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 321 k 2024-05-17T14:49:28.811 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 300 k 2024-05-17T14:49:28.811 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 99 k 2024-05-17T14:49:28.811 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-17T14:49:28.811 INFO:teuthology.orchestra.run.smithi195.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-17T14:49:28.811 INFO:teuthology.orchestra.run.smithi195.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-17T14:49:28.811 INFO:teuthology.orchestra.run.smithi195.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-17T14:49:28.811 INFO:teuthology.orchestra.run.smithi195.stdout:Installing weak dependencies: 2024-05-17T14:49:28.811 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-17T14:49:28.811 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:49:28.811 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-17T14:49:28.811 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:49:28.811 INFO:teuthology.orchestra.run.smithi195.stdout:Install 30 Packages 2024-05-17T14:49:28.812 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrade 2 Packages 2024-05-17T14:49:28.812 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:49:28.812 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 60 M 2024-05-17T14:49:28.812 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-17T14:49:30.029 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-05-17T14:49:30.303 INFO:teuthology.orchestra.run.smithi195.stdout:(1/32): ceph-base-19.0.0-3735.gee59d86a.el9.x86 6.2 MB/s | 5.4 MB 00:00 2024-05-17T14:49:30.412 INFO:teuthology.orchestra.run.smithi195.stdout:(2/32): ceph-selinux-19.0.0-3735.gee59d86a.el9. 232 kB/s | 25 kB 00:00 2024-05-17T14:49:30.570 INFO:teuthology.orchestra.run.smithi195.stdout:(3/32): libcephfs2-19.0.0-3735.gee59d86a.el9.x8 4.6 MB/s | 748 kB 00:00 2024-05-17T14:49:30.679 INFO:teuthology.orchestra.run.smithi195.stdout:(4/32): libradosstriper1-19.0.0-3735.gee59d86a. 4.5 MB/s | 501 kB 00:00 2024-05-17T14:49:31.296 INFO:teuthology.orchestra.run.smithi195.stdout:(5/32): librgw2-19.0.0-3735.gee59d86a.el9.x86_6 8.7 MB/s | 5.4 MB 00:00 2024-05-17T14:49:31.421 INFO:teuthology.orchestra.run.smithi195.stdout:(6/32): ceph-radosgw-19.0.0-3735.gee59d86a.el9. 5.3 MB/s | 11 MB 00:01 2024-05-17T14:49:31.438 INFO:teuthology.orchestra.run.smithi195.stdout:(7/32): python3-ceph-argparse-19.0.0-3735.gee59 317 kB/s | 45 kB 00:00 2024-05-17T14:49:31.647 INFO:teuthology.orchestra.run.smithi195.stdout:(8/32): ceph-common-19.0.0-3735.gee59d86a.el9.x 9.6 MB/s | 21 MB 00:02 2024-05-17T14:49:31.663 INFO:teuthology.orchestra.run.smithi195.stdout:(9/32): python3-ceph-common-19.0.0-3735.gee59d8 560 kB/s | 136 kB 00:00 2024-05-17T14:49:31.689 INFO:teuthology.orchestra.run.smithi195.stdout:(10/32): python3-cephfs-19.0.0-3735.gee59d86a.e 646 kB/s | 162 kB 00:00 2024-05-17T14:49:31.731 INFO:teuthology.orchestra.run.smithi195.stdout:(11/32): python3-rados-19.0.0-3735.gee59d86a.el 3.8 MB/s | 321 kB 00:00 2024-05-17T14:49:31.747 INFO:teuthology.orchestra.run.smithi195.stdout:(12/32): python3-rgw-19.0.0-3735.gee59d86a.el9. 1.7 MB/s | 99 kB 00:00 2024-05-17T14:49:31.781 INFO:teuthology.orchestra.run.smithi195.stdout:(13/32): python3-rbd-19.0.0-3735.gee59d86a.el9. 2.5 MB/s | 300 kB 00:00 2024-05-17T14:49:31.823 INFO:teuthology.orchestra.run.smithi195.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 1.1 MB/s | 107 kB 00:00 2024-05-17T14:49:31.848 INFO:teuthology.orchestra.run.smithi195.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 677 kB/s | 45 kB 00:00 2024-05-17T14:49:31.915 INFO:teuthology.orchestra.run.smithi195.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 957 kB/s | 160 kB 00:00 2024-05-17T14:49:31.974 INFO:teuthology.orchestra.run.smithi195.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 724 kB/s | 42 kB 00:00 2024-05-17T14:49:31.999 INFO:teuthology.orchestra.run.smithi195.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-05-17T14:49:32.024 INFO:teuthology.orchestra.run.smithi195.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.2 MB/s | 662 kB 00:00 2024-05-17T14:49:32.067 INFO:teuthology.orchestra.run.smithi195.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 534 kB/s | 36 kB 00:00 2024-05-17T14:49:32.092 INFO:teuthology.orchestra.run.smithi195.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 703 kB/s | 47 kB 00:00 2024-05-17T14:49:32.117 INFO:teuthology.orchestra.run.smithi195.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.5 MB/s | 221 kB 00:00 2024-05-17T14:49:32.193 INFO:teuthology.orchestra.run.smithi195.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-05-17T14:49:32.226 INFO:teuthology.orchestra.run.smithi195.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.9 MB/s | 309 kB 00:00 2024-05-17T14:49:32.251 INFO:teuthology.orchestra.run.smithi195.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 830 kB/s | 49 kB 00:00 2024-05-17T14:49:32.277 INFO:teuthology.orchestra.run.smithi195.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-05-17T14:49:32.319 INFO:teuthology.orchestra.run.smithi195.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-05-17T14:49:32.336 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:49:32.348 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:49:32.348 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-05-17T14:49:32.348 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:49:32.348 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-05-17T14:49:32.348 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 11 M 2024-05-17T14:49:32.349 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2024-05-17T14:49:32.349 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 3.4 M 2024-05-17T14:49:32.349 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 3.1 M 2024-05-17T14:49:32.349 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-05-17T14:49:32.349 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-17T14:49:32.349 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 5.4 M 2024-05-17T14:49:32.349 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 21 M 2024-05-17T14:49:32.349 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 25 k 2024-05-17T14:49:32.349 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-17T14:49:32.349 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-17T14:49:32.349 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-17T14:49:32.349 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 748 k 2024-05-17T14:49:32.350 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-17T14:49:32.350 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-17T14:49:32.350 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-17T14:49:32.350 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 501 k 2024-05-17T14:49:32.350 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-17T14:49:32.350 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 5.4 M 2024-05-17T14:49:32.350 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-17T14:49:32.350 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-17T14:49:32.350 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-17T14:49:32.350 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 45 k 2024-05-17T14:49:32.350 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 136 k 2024-05-17T14:49:32.350 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 162 k 2024-05-17T14:49:32.351 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-17T14:49:32.351 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 321 k 2024-05-17T14:49:32.351 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 300 k 2024-05-17T14:49:32.351 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 99 k 2024-05-17T14:49:32.351 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-17T14:49:32.351 INFO:teuthology.orchestra.run.smithi134.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-17T14:49:32.351 INFO:teuthology.orchestra.run.smithi134.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-17T14:49:32.351 INFO:teuthology.orchestra.run.smithi134.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-17T14:49:32.351 INFO:teuthology.orchestra.run.smithi134.stdout:Installing weak dependencies: 2024-05-17T14:49:32.351 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-17T14:49:32.351 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:49:32.351 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-05-17T14:49:32.351 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:49:32.351 INFO:teuthology.orchestra.run.smithi134.stdout:Install 30 Packages 2024-05-17T14:49:32.351 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 2 Packages 2024-05-17T14:49:32.352 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:49:32.352 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 60 M 2024-05-17T14:49:32.352 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-05-17T14:49:32.369 INFO:teuthology.orchestra.run.smithi195.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-05-17T14:49:32.444 INFO:teuthology.orchestra.run.smithi195.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-05-17T14:49:32.486 INFO:teuthology.orchestra.run.smithi195.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-05-17T14:49:32.670 INFO:teuthology.orchestra.run.smithi195.stdout:(31/32): librados2-19.0.0-3735.gee59d86a.el9.x8 11 MB/s | 3.4 MB 00:00 2024-05-17T14:49:32.814 INFO:teuthology.orchestra.run.smithi195.stdout:(32/32): librbd1-19.0.0-3735.gee59d86a.el9.x86_ 8.6 MB/s | 3.1 MB 00:00 2024-05-17T14:49:32.814 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:49:32.815 INFO:teuthology.orchestra.run.smithi195.stdout:Total 15 MB/s | 60 MB 00:04 2024-05-17T14:49:32.924 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-17T14:49:32.959 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-17T14:49:32.959 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-17T14:49:33.497 INFO:teuthology.orchestra.run.smithi134.stdout:(1/32): ceph-base-19.0.0-3735.gee59d86a.el9.x86 5.4 MB/s | 5.4 MB 00:00 2024-05-17T14:49:33.571 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-17T14:49:33.571 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-17T14:49:33.622 INFO:teuthology.orchestra.run.smithi134.stdout:(2/32): ceph-radosgw-19.0.0-3735.gee59d86a.el9. 9.4 MB/s | 11 MB 00:01 2024-05-17T14:49:33.647 INFO:teuthology.orchestra.run.smithi134.stdout:(3/32): ceph-selinux-19.0.0-3735.gee59d86a.el9. 167 kB/s | 25 kB 00:00 2024-05-17T14:49:33.697 INFO:teuthology.orchestra.run.smithi134.stdout:(4/32): libcephfs2-19.0.0-3735.gee59d86a.el9.x8 9.7 MB/s | 748 kB 00:00 2024-05-17T14:49:33.731 INFO:teuthology.orchestra.run.smithi134.stdout:(5/32): libradosstriper1-19.0.0-3735.gee59d86a. 5.9 MB/s | 501 kB 00:00 2024-05-17T14:49:33.848 INFO:teuthology.orchestra.run.smithi134.stdout:(6/32): python3-ceph-argparse-19.0.0-3735.gee59 386 kB/s | 45 kB 00:00 2024-05-17T14:49:33.948 INFO:teuthology.orchestra.run.smithi134.stdout:(7/32): python3-ceph-common-19.0.0-3735.gee59d8 1.3 MB/s | 136 kB 00:00 2024-05-17T14:49:34.073 INFO:teuthology.orchestra.run.smithi134.stdout:(8/32): librgw2-19.0.0-3735.gee59d86a.el9.x86_6 14 MB/s | 5.4 MB 00:00 2024-05-17T14:49:34.099 INFO:teuthology.orchestra.run.smithi134.stdout:(9/32): python3-cephfs-19.0.0-3735.gee59d86a.el 1.1 MB/s | 162 kB 00:00 2024-05-17T14:49:34.141 INFO:teuthology.orchestra.run.smithi134.stdout:(10/32): python3-rados-19.0.0-3735.gee59d86a.el 4.7 MB/s | 321 kB 00:00 2024-05-17T14:49:34.174 INFO:teuthology.orchestra.run.smithi134.stdout:(11/32): python3-rbd-19.0.0-3735.gee59d86a.el9. 3.9 MB/s | 300 kB 00:00 2024-05-17T14:49:34.208 INFO:teuthology.orchestra.run.smithi134.stdout:(12/32): python3-rgw-19.0.0-3735.gee59d86a.el9. 1.5 MB/s | 99 kB 00:00 2024-05-17T14:49:34.308 INFO:teuthology.orchestra.run.smithi134.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 799 kB/s | 107 kB 00:00 2024-05-17T14:49:34.333 INFO:teuthology.orchestra.run.smithi134.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-05-17T14:49:34.359 INFO:teuthology.orchestra.run.smithi134.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 899 kB/s | 45 kB 00:00 2024-05-17T14:49:34.417 INFO:teuthology.orchestra.run.smithi134.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 662 kB 00:00 2024-05-17T14:49:34.445 INFO:teuthology.orchestra.run.smithi134.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-05-17T14:49:34.468 INFO:teuthology.orchestra.run.smithi134.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 841 kB/s | 42 kB 00:00 2024-05-17T14:49:34.493 INFO:teuthology.orchestra.run.smithi134.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-05-17T14:49:34.518 INFO:teuthology.orchestra.run.smithi134.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 717 kB/s | 36 kB 00:00 2024-05-17T14:49:34.544 INFO:teuthology.orchestra.run.smithi134.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-05-17T14:49:34.794 INFO:teuthology.orchestra.run.smithi134.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-05-17T14:49:34.844 INFO:teuthology.orchestra.run.smithi134.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 523 kB/s | 26 kB 00:00 2024-05-17T14:49:34.894 INFO:teuthology.orchestra.run.smithi134.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 988 kB/s | 49 kB 00:00 2024-05-17T14:49:34.978 INFO:teuthology.orchestra.run.smithi134.stdout:(25/32): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-05-17T14:49:34.993 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-17T14:49:35.003 INFO:teuthology.orchestra.run.smithi134.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 633 kB/s | 67 kB 00:00 2024-05-17T14:49:35.054 INFO:teuthology.orchestra.run.smithi134.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-05-17T14:49:35.063 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-05-17T14:49:35.087 INFO:teuthology.orchestra.run.smithi134.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-05-17T14:49:35.092 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-05-17T14:49:35.125 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-05-17T14:49:35.138 INFO:teuthology.orchestra.run.smithi134.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-05-17T14:49:35.148 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-17T14:49:35.173 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-17T14:49:35.367 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-17T14:49:35.422 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-17T14:49:35.580 INFO:teuthology.orchestra.run.smithi134.stdout:(30/32): librbd1-19.0.0-3735.gee59d86a.el9.x86_ 7.1 MB/s | 3.1 MB 00:00 2024-05-17T14:49:35.639 INFO:teuthology.orchestra.run.smithi134.stdout:(31/32): librados2-19.0.0-3735.gee59d86a.el9.x8 6.2 MB/s | 3.4 MB 00:00 2024-05-17T14:49:35.798 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-05-17T14:49:35.877 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librados2-2:19.0.0-3735.gee59d86a.el9.x86_64 7/34 2024-05-17T14:49:35.953 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:19.0.0-3735.gee59d86a.el9.x86_64 7/34 2024-05-17T14:49:36.042 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rados-2:19.0.0-3735.gee59d86a.el9.x86_64 8/34 2024-05-17T14:49:36.070 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs2-2:19.0.0-3735.gee59d86a.el9.x86_64 9/34 2024-05-17T14:49:36.142 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs2-2:19.0.0-3735.gee59d86a.el9.x86_64 9/34 2024-05-17T14:49:36.170 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-05-17T14:49:36.214 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-argparse-2:19.0.0-3735.gee59d86a.el9. 11/34 2024-05-17T14:49:36.284 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cephfs-2:19.0.0-3735.gee59d86a.el9.x86_64 12/34 2024-05-17T14:49:36.297 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libradosstriper1-2:19.0.0-3735.gee59d86a.el9.x86_6 13/34 2024-05-17T14:49:36.362 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3735.gee59d86a.el9.x86_6 13/34 2024-05-17T14:49:36.388 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-05-17T14:49:36.447 INFO:teuthology.orchestra.run.smithi134.stdout:(32/32): ceph-common-19.0.0-3735.gee59d86a.el9. 5.4 MB/s | 21 MB 00:03 2024-05-17T14:49:36.452 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:49:36.453 INFO:teuthology.orchestra.run.smithi134.stdout:Total 15 MB/s | 60 MB 00:04 2024-05-17T14:49:36.509 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-05-17T14:49:36.539 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-05-17T14:49:36.553 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-05-17T14:49:36.580 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-05-17T14:49:36.589 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-05-17T14:49:36.589 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-05-17T14:49:36.612 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-05-17T14:49:36.641 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-05-17T14:49:36.666 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-05-17T14:49:36.985 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-05-17T14:49:37.020 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librbd1-2:19.0.0-3735.gee59d86a.el9.x86_64 22/34 2024-05-17T14:49:37.111 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:19.0.0-3735.gee59d86a.el9.x86_64 22/34 2024-05-17T14:49:37.142 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rbd-2:19.0.0-3735.gee59d86a.el9.x86_64 23/34 2024-05-17T14:49:37.178 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-05-17T14:49:37.198 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-05-17T14:49:37.198 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-05-17T14:49:37.661 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-05-17T14:49:37.690 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librgw2-2:19.0.0-3735.gee59d86a.el9.x86_64 26/34 2024-05-17T14:49:37.769 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librgw2-2:19.0.0-3735.gee59d86a.el9.x86_64 26/34 2024-05-17T14:49:37.846 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rgw-2:19.0.0-3735.gee59d86a.el9.x86_64 27/34 2024-05-17T14:49:37.876 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-common-2:19.0.0-3735.gee59d86a.el9.x8 28/34 2024-05-17T14:49:38.621 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-05-17T14:49:38.697 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-05-17T14:49:38.752 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-05-17T14:49:38.777 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-05-17T14:49:38.800 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-17T14:49:38.824 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-17T14:49:39.019 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-17T14:49:39.073 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-17T14:49:39.418 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-05-17T14:49:39.529 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:19.0.0-3735.gee59d86a.el9.x86_64 7/34 2024-05-17T14:49:39.630 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:19.0.0-3735.gee59d86a.el9.x86_64 7/34 2024-05-17T14:49:39.719 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:19.0.0-3735.gee59d86a.el9.x86_64 8/34 2024-05-17T14:49:39.749 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:19.0.0-3735.gee59d86a.el9.x86_64 9/34 2024-05-17T14:49:39.801 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:19.0.0-3735.gee59d86a.el9.x86_64 9/34 2024-05-17T14:49:39.830 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-05-17T14:49:39.873 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:19.0.0-3735.gee59d86a.el9. 11/34 2024-05-17T14:49:39.942 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:19.0.0-3735.gee59d86a.el9.x86_64 12/34 2024-05-17T14:49:39.966 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:19.0.0-3735.gee59d86a.el9.x86_6 13/34 2024-05-17T14:49:40.030 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3735.gee59d86a.el9.x86_6 13/34 2024-05-17T14:49:40.056 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-05-17T14:49:40.164 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-05-17T14:49:40.166 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:19.0.0-3735.gee59d86a.el9.x86_64 29/34 2024-05-17T14:49:40.198 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-common-2:19.0.0-3735.gee59d86a.el9.x86_64 29/34 2024-05-17T14:49:40.207 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-05-17T14:49:40.248 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-05-17T14:49:40.288 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-05-17T14:49:40.317 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-05-17T14:49:40.350 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-05-17T14:49:40.666 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-05-17T14:49:40.705 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:19.0.0-3735.gee59d86a.el9.x86_64 22/34 2024-05-17T14:49:40.758 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:19.0.0-3735.gee59d86a.el9.x86_64 29/34 2024-05-17T14:49:40.793 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-base-2:19.0.0-3735.gee59d86a.el9.x86_64 30/34 2024-05-17T14:49:40.794 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:19.0.0-3735.gee59d86a.el9.x86_64 22/34 2024-05-17T14:49:40.826 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:19.0.0-3735.gee59d86a.el9.x86_64 23/34 2024-05-17T14:49:40.862 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-05-17T14:49:40.866 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-base-2:19.0.0-3735.gee59d86a.el9.x86_64 30/34 2024-05-17T14:49:40.866 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-17T14:49:40.866 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-17T14:49:40.866 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:49:40.895 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-selinux-2:19.0.0-3735.gee59d86a.el9.x86_64 31/34 2024-05-17T14:49:41.346 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-05-17T14:49:41.383 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:19.0.0-3735.gee59d86a.el9.x86_64 26/34 2024-05-17T14:49:41.454 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:19.0.0-3735.gee59d86a.el9.x86_64 26/34 2024-05-17T14:49:41.523 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:19.0.0-3735.gee59d86a.el9.x86_64 27/34 2024-05-17T14:49:41.552 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:19.0.0-3735.gee59d86a.el9.x8 28/34 2024-05-17T14:49:43.886 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-3735.gee59d86a.el9.x86_64 29/34 2024-05-17T14:49:43.923 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:19.0.0-3735.gee59d86a.el9.x86_64 29/34 2024-05-17T14:49:44.487 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-3735.gee59d86a.el9.x86_64 29/34 2024-05-17T14:49:44.527 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:19.0.0-3735.gee59d86a.el9.x86_64 30/34 2024-05-17T14:49:44.603 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:19.0.0-3735.gee59d86a.el9.x86_64 30/34 2024-05-17T14:49:44.603 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-17T14:49:44.603 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-17T14:49:44.603 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:49:44.637 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:19.0.0-3735.gee59d86a.el9.x86_64 31/34 2024-05-17T14:49:50.433 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3735.gee59d86a.el9.x86_64 31/34 2024-05-17T14:49:50.433 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /sys 2024-05-17T14:49:50.433 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /proc 2024-05-17T14:49:50.433 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /mnt 2024-05-17T14:49:50.433 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /var/tmp 2024-05-17T14:49:50.433 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /home 2024-05-17T14:49:50.433 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /root 2024-05-17T14:49:50.433 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /tmp 2024-05-17T14:49:50.433 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:49:51.430 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-radosgw-2:19.0.0-3735.gee59d86a.el9.x86_64 32/34 2024-05-17T14:49:51.456 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3735.gee59d86a.el9.x86_64 32/34 2024-05-17T14:49:51.456 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-17T14:49:51.457 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-17T14:49:51.457 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-17T14:49:51.457 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-17T14:49:51.457 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:49:51.458 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-17T14:49:51.549 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-17T14:49:51.550 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-17T14:49:52.725 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-17T14:49:52.725 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-base-2:19.0.0-3735.gee59d86a.el9.x86_64 1/34 2024-05-17T14:49:52.726 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-common-2:19.0.0-3735.gee59d86a.el9.x86_64 2/34 2024-05-17T14:49:52.726 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-radosgw-2:19.0.0-3735.gee59d86a.el9.x86_64 3/34 2024-05-17T14:49:52.726 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-selinux-2:19.0.0-3735.gee59d86a.el9.x86_64 4/34 2024-05-17T14:49:52.726 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs2-2:19.0.0-3735.gee59d86a.el9.x86_64 5/34 2024-05-17T14:49:52.726 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libradosstriper1-2:19.0.0-3735.gee59d86a.el9.x86_6 6/34 2024-05-17T14:49:52.726 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librgw2-2:19.0.0-3735.gee59d86a.el9.x86_64 7/34 2024-05-17T14:49:52.726 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3735.gee59d86a.el9. 8/34 2024-05-17T14:49:52.726 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-common-2:19.0.0-3735.gee59d86a.el9.x8 9/34 2024-05-17T14:49:52.726 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cephfs-2:19.0.0-3735.gee59d86a.el9.x86_64 10/34 2024-05-17T14:49:52.726 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rados-2:19.0.0-3735.gee59d86a.el9.x86_64 11/34 2024-05-17T14:49:52.726 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rbd-2:19.0.0-3735.gee59d86a.el9.x86_64 12/34 2024-05-17T14:49:52.726 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rgw-2:19.0.0-3735.gee59d86a.el9.x86_64 13/34 2024-05-17T14:49:52.727 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-05-17T14:49:52.727 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-05-17T14:49:52.727 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-05-17T14:49:52.727 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-05-17T14:49:52.727 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-05-17T14:49:52.727 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-05-17T14:49:52.727 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-05-17T14:49:52.727 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-05-17T14:49:52.727 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-05-17T14:49:52.727 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-05-17T14:49:52.727 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-05-17T14:49:52.727 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-05-17T14:49:52.727 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-05-17T14:49:52.728 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-05-17T14:49:52.728 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-05-17T14:49:52.728 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-05-17T14:49:52.728 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-05-17T14:49:52.728 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:19.0.0-3735.gee59d86a.el9.x86_64 31/34 2024-05-17T14:49:52.728 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-05-17T14:49:52.729 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:19.0.0-3735.gee59d86a.el9.x86_64 33/34 2024-05-17T14:49:53.487 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-05-17T14:49:53.487 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:49:53.487 INFO:teuthology.orchestra.run.smithi195.stdout:Upgraded: 2024-05-17T14:49:53.487 INFO:teuthology.orchestra.run.smithi195.stdout: librados2-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:53.487 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:53.488 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-17T14:49:53.488 INFO:teuthology.orchestra.run.smithi195.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-17T14:49:53.488 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:53.488 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:53.488 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:53.488 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:53.488 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-17T14:49:53.488 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-17T14:49:53.488 INFO:teuthology.orchestra.run.smithi195.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-17T14:49:53.488 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:53.489 INFO:teuthology.orchestra.run.smithi195.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-17T14:49:53.489 INFO:teuthology.orchestra.run.smithi195.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-17T14:49:53.489 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-17T14:49:53.489 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:53.489 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-17T14:49:53.489 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:53.489 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-17T14:49:53.489 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-17T14:49:53.489 INFO:teuthology.orchestra.run.smithi195.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-17T14:49:53.489 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:53.489 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:53.489 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:53.489 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-17T14:49:53.489 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:53.489 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:53.490 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:53.490 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-17T14:49:53.490 INFO:teuthology.orchestra.run.smithi195.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-17T14:49:53.490 INFO:teuthology.orchestra.run.smithi195.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-17T14:49:53.490 INFO:teuthology.orchestra.run.smithi195.stdout: re2-1:20211101-3.el9.x86_64 2024-05-17T14:49:53.490 INFO:teuthology.orchestra.run.smithi195.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-17T14:49:53.490 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:49:53.490 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:49:53.883 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-test 2024-05-17T14:49:54.430 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3735.gee59d86a.el9.x86_64 31/34 2024-05-17T14:49:54.431 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2024-05-17T14:49:54.431 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2024-05-17T14:49:54.431 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2024-05-17T14:49:54.431 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2024-05-17T14:49:54.431 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2024-05-17T14:49:54.431 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /root 2024-05-17T14:49:54.431 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2024-05-17T14:49:54.431 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:49:54.458 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:28 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:49:54.579 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:49:54.580 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:49:54.580 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-17T14:49:54.580 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:49:54.580 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-17T14:49:54.580 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 49 M 2024-05-17T14:49:54.580 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-17T14:49:54.580 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 163 k 2024-05-17T14:49:54.581 INFO:teuthology.orchestra.run.smithi195.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-17T14:49:54.581 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-17T14:49:54.581 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:49:54.581 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-17T14:49:54.581 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:49:54.581 INFO:teuthology.orchestra.run.smithi195.stdout:Install 4 Packages 2024-05-17T14:49:54.581 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:49:54.581 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 50 M 2024-05-17T14:49:54.582 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 211 M 2024-05-17T14:49:54.582 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-17T14:49:54.987 INFO:teuthology.orchestra.run.smithi195.stdout:(1/4): libcephsqlite-19.0.0-3735.gee59d86a.el9. 623 kB/s | 163 kB 00:00 2024-05-17T14:49:55.037 INFO:teuthology.orchestra.run.smithi195.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 976 kB/s | 305 kB 00:00 2024-05-17T14:49:55.137 INFO:teuthology.orchestra.run.smithi195.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 425 kB/s | 64 kB 00:00 2024-05-17T14:49:55.406 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:19.0.0-3735.gee59d86a.el9.x86_64 32/34 2024-05-17T14:49:55.430 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3735.gee59d86a.el9.x86_64 32/34 2024-05-17T14:49:55.431 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-17T14:49:55.431 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-17T14:49:55.431 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-17T14:49:55.431 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-17T14:49:55.431 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:49:55.432 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-17T14:49:55.508 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-17T14:49:55.508 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-17T14:49:56.637 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-17T14:49:56.637 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:19.0.0-3735.gee59d86a.el9.x86_64 1/34 2024-05-17T14:49:56.637 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:19.0.0-3735.gee59d86a.el9.x86_64 2/34 2024-05-17T14:49:56.637 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:19.0.0-3735.gee59d86a.el9.x86_64 3/34 2024-05-17T14:49:56.637 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:19.0.0-3735.gee59d86a.el9.x86_64 4/34 2024-05-17T14:49:56.638 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:19.0.0-3735.gee59d86a.el9.x86_64 5/34 2024-05-17T14:49:56.638 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:19.0.0-3735.gee59d86a.el9.x86_6 6/34 2024-05-17T14:49:56.638 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:19.0.0-3735.gee59d86a.el9.x86_64 7/34 2024-05-17T14:49:56.638 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3735.gee59d86a.el9. 8/34 2024-05-17T14:49:56.638 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:19.0.0-3735.gee59d86a.el9.x8 9/34 2024-05-17T14:49:56.638 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:19.0.0-3735.gee59d86a.el9.x86_64 10/34 2024-05-17T14:49:56.638 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:19.0.0-3735.gee59d86a.el9.x86_64 11/34 2024-05-17T14:49:56.638 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:19.0.0-3735.gee59d86a.el9.x86_64 12/34 2024-05-17T14:49:56.638 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:19.0.0-3735.gee59d86a.el9.x86_64 13/34 2024-05-17T14:49:56.638 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-05-17T14:49:56.638 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-05-17T14:49:56.638 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-05-17T14:49:56.639 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-05-17T14:49:56.639 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-05-17T14:49:56.639 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-05-17T14:49:56.639 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-05-17T14:49:56.639 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-05-17T14:49:56.639 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-05-17T14:49:56.639 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-05-17T14:49:56.639 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-05-17T14:49:56.639 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-05-17T14:49:56.639 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-05-17T14:49:56.639 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-05-17T14:49:56.639 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-05-17T14:49:56.639 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-05-17T14:49:56.640 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-05-17T14:49:56.640 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:19.0.0-3735.gee59d86a.el9.x86_64 31/34 2024-05-17T14:49:56.640 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-05-17T14:49:56.640 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:19.0.0-3735.gee59d86a.el9.x86_64 33/34 2024-05-17T14:49:57.584 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-05-17T14:49:57.584 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:49:57.584 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2024-05-17T14:49:57.585 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:57.585 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:57.585 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-05-17T14:49:57.585 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-17T14:49:57.585 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:57.585 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:57.585 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:57.585 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:57.585 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-17T14:49:57.585 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-17T14:49:57.585 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-17T14:49:57.585 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:57.585 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-17T14:49:57.586 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-17T14:49:57.586 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-17T14:49:57.586 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:57.586 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-17T14:49:57.586 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:57.586 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-17T14:49:57.586 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-17T14:49:57.586 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-17T14:49:57.586 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:57.586 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:57.587 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:57.587 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-17T14:49:57.587 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:57.587 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:57.587 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:49:57.587 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-17T14:49:57.587 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-17T14:49:57.587 INFO:teuthology.orchestra.run.smithi134.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-17T14:49:57.587 INFO:teuthology.orchestra.run.smithi134.stdout: re2-1:20211101-3.el9.x86_64 2024-05-17T14:49:57.587 INFO:teuthology.orchestra.run.smithi134.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-17T14:49:57.587 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:49:57.587 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:49:57.613 INFO:teuthology.orchestra.run.smithi195.stdout:(4/4): ceph-test-19.0.0-3735.gee59d86a.el9.x86_ 17 MB/s | 49 MB 00:02 2024-05-17T14:49:57.615 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:49:57.615 INFO:teuthology.orchestra.run.smithi195.stdout:Total 16 MB/s | 50 MB 00:03 2024-05-17T14:49:57.644 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-17T14:49:57.664 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-17T14:49:57.664 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-17T14:49:57.982 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test 2024-05-17T14:49:58.035 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-17T14:49:58.035 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-17T14:49:58.558 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:29 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:49:58.560 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-17T14:49:58.639 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-17T14:49:58.680 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:49:58.680 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:49:58.680 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-05-17T14:49:58.680 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:49:58.680 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-05-17T14:49:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 49 M 2024-05-17T14:49:58.681 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-05-17T14:49:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 163 k 2024-05-17T14:49:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-17T14:49:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-17T14:49:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:49:58.681 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-05-17T14:49:58.681 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:49:58.681 INFO:teuthology.orchestra.run.smithi134.stdout:Install 4 Packages 2024-05-17T14:49:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:49:58.682 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 50 M 2024-05-17T14:49:58.682 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 211 M 2024-05-17T14:49:58.682 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-05-17T14:49:58.686 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-17T14:49:58.714 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephsqlite-2:19.0.0-3735.gee59d86a.el9.x86_64 3/4 2024-05-17T14:49:58.979 INFO:teuthology.orchestra.run.smithi134.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 305 kB 00:00 2024-05-17T14:49:59.021 INFO:teuthology.orchestra.run.smithi134.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-05-17T14:49:59.071 INFO:teuthology.orchestra.run.smithi134.stdout:(3/4): libcephsqlite-19.0.0-3735.gee59d86a.el9. 625 kB/s | 163 kB 00:00 2024-05-17T14:50:01.189 INFO:teuthology.orchestra.run.smithi134.stdout:(4/4): ceph-test-19.0.0-3735.gee59d86a.el9.x86_ 21 MB/s | 49 MB 00:02 2024-05-17T14:50:01.190 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:50:01.191 INFO:teuthology.orchestra.run.smithi134.stdout:Total 20 MB/s | 50 MB 00:02 2024-05-17T14:50:01.218 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-05-17T14:50:01.239 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-05-17T14:50:01.239 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-05-17T14:50:01.610 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-05-17T14:50:01.610 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-05-17T14:50:02.147 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-05-17T14:50:02.223 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-17T14:50:02.262 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-17T14:50:02.290 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:19.0.0-3735.gee59d86a.el9.x86_64 3/4 2024-05-17T14:50:03.379 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3735.gee59d86a.el9.x86_64 3/4 2024-05-17T14:50:03.455 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-test-2:19.0.0-3735.gee59d86a.el9.x86_64 4/4 2024-05-17T14:50:05.856 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-test-2:19.0.0-3735.gee59d86a.el9.x86_64 4/4 2024-05-17T14:50:05.857 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-test-2:19.0.0-3735.gee59d86a.el9.x86_64 1/4 2024-05-17T14:50:05.857 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephsqlite-2:19.0.0-3735.gee59d86a.el9.x86_64 2/4 2024-05-17T14:50:05.857 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-17T14:50:06.144 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-17T14:50:06.144 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:06.144 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-17T14:50:06.144 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:50:06.144 INFO:teuthology.orchestra.run.smithi195.stdout: libcephsqlite-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:50:06.144 INFO:teuthology.orchestra.run.smithi195.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-17T14:50:06.144 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-17T14:50:06.144 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:06.145 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:50:06.414 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph 2024-05-17T14:50:07.007 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:41 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:50:07.018 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3735.gee59d86a.el9.x86_64 3/4 2024-05-17T14:50:07.139 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:50:07.142 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:50:07.142 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-17T14:50:07.142 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:50:07.142 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-17T14:50:07.142 INFO:teuthology.orchestra.run.smithi195.stdout: ceph x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 6.6 k 2024-05-17T14:50:07.142 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-17T14:50:07.142 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 2.4 M 2024-05-17T14:50:07.142 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 1.5 M 2024-05-17T14:50:07.142 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 274 k 2024-05-17T14:50:07.143 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 4.7 M 2024-05-17T14:50:07.143 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 17 M 2024-05-17T14:50:07.143 INFO:teuthology.orchestra.run.smithi195.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-17T14:50:07.143 INFO:teuthology.orchestra.run.smithi195.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-17T14:50:07.143 INFO:teuthology.orchestra.run.smithi195.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-17T14:50:07.143 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-17T14:50:07.143 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-17T14:50:07.143 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-17T14:50:07.143 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-17T14:50:07.143 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-17T14:50:07.143 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-17T14:50:07.143 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-17T14:50:07.143 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-collections 2024-05-17T14:50:07.143 INFO:teuthology.orchestra.run.smithi195.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-05-17T14:50:07.143 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-17T14:50:07.144 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-17T14:50:07.144 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-17T14:50:07.144 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-17T14:50:07.144 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-17T14:50:07.144 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-17T14:50:07.144 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-17T14:50:07.144 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-17T14:50:07.144 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-17T14:50:07.144 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-17T14:50:07.144 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-17T14:50:07.144 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-17T14:50:07.144 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-17T14:50:07.144 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-17T14:50:07.144 INFO:teuthology.orchestra.run.smithi195.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-17T14:50:07.144 INFO:teuthology.orchestra.run.smithi195.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-17T14:50:07.145 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-17T14:50:07.145 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-17T14:50:07.145 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-17T14:50:07.145 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:07.145 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-17T14:50:07.145 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:50:07.145 INFO:teuthology.orchestra.run.smithi195.stdout:Install 36 Packages 2024-05-17T14:50:07.145 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:07.146 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 30 M 2024-05-17T14:50:07.146 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 108 M 2024-05-17T14:50:07.146 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-17T14:50:07.190 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:19.0.0-3735.gee59d86a.el9.x86_64 4/4 2024-05-17T14:50:08.174 INFO:teuthology.orchestra.run.smithi195.stdout:(1/36): ceph-19.0.0-3735.gee59d86a.el9.x86_64.r 16 kB/s | 6.6 kB 00:00 2024-05-17T14:50:09.182 INFO:teuthology.orchestra.run.smithi195.stdout:(2/36): ceph-mon-19.0.0-3735.gee59d86a.el9.x86_ 4.7 MB/s | 4.7 MB 00:01 2024-05-17T14:50:09.529 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-2:19.0.0-3735.gee59d86a.el9.x86_64 4/4 2024-05-17T14:50:09.529 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:19.0.0-3735.gee59d86a.el9.x86_64 1/4 2024-05-17T14:50:09.529 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:19.0.0-3735.gee59d86a.el9.x86_64 2/4 2024-05-17T14:50:09.530 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-17T14:50:09.574 INFO:teuthology.orchestra.run.smithi195.stdout:(3/36): ceph-mds-19.0.0-3735.gee59d86a.el9.x86_ 1.3 MB/s | 2.4 MB 00:01 2024-05-17T14:50:09.608 INFO:teuthology.orchestra.run.smithi195.stdout:(4/36): ceph-mgr-19.0.0-3735.gee59d86a.el9.x86_ 846 kB/s | 1.5 MB 00:01 2024-05-17T14:50:09.733 INFO:teuthology.orchestra.run.smithi195.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-05-17T14:50:09.758 INFO:teuthology.orchestra.run.smithi195.stdout:(6/36): ceph-mgr-modules-core-19.0.0-3735.gee59 1.5 MB/s | 274 kB 00:00 2024-05-17T14:50:09.800 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-17T14:50:09.800 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:09.800 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-05-17T14:50:09.800 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:50:09.800 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:50:09.800 INFO:teuthology.orchestra.run.smithi134.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-17T14:50:09.800 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-17T14:50:09.800 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:09.800 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:50:09.826 INFO:teuthology.orchestra.run.smithi195.stdout:(7/36): python3-cryptography-36.0.1-4.el9.x86_6 13 MB/s | 1.2 MB 00:00 2024-05-17T14:50:09.867 INFO:teuthology.orchestra.run.smithi195.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-05-17T14:50:09.892 INFO:teuthology.orchestra.run.smithi195.stdout:(9/36): python3-ply-3.11-14.el9.noarch.rpm 796 kB/s | 106 kB 00:00 2024-05-17T14:50:09.918 INFO:teuthology.orchestra.run.smithi195.stdout:(10/36): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-05-17T14:50:09.960 INFO:teuthology.orchestra.run.smithi195.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-05-17T14:50:10.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph 2024-05-17T14:50:10.127 INFO:teuthology.orchestra.run.smithi195.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 904 kB/s | 188 kB 00:00 2024-05-17T14:50:10.152 INFO:teuthology.orchestra.run.smithi195.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 897 kB/s | 172 kB 00:00 2024-05-17T14:50:10.185 INFO:teuthology.orchestra.run.smithi195.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 595 kB/s | 35 kB 00:00 2024-05-17T14:50:10.211 INFO:teuthology.orchestra.run.smithi195.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 871 kB/s | 51 kB 00:00 2024-05-17T14:50:10.244 INFO:teuthology.orchestra.run.smithi195.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 711 kB/s | 42 kB 00:00 2024-05-17T14:50:10.478 INFO:teuthology.orchestra.run.smithi195.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 83 kB/s | 22 kB 00:00 2024-05-17T14:50:10.669 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:41 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:50:10.728 INFO:teuthology.orchestra.run.smithi195.stdout:(18/36): ceph-osd-19.0.0-3735.gee59d86a.el9.x86 11 MB/s | 17 MB 00:01 2024-05-17T14:50:10.753 INFO:teuthology.orchestra.run.smithi195.stdout:(19/36): luarocks-3.9.2-5.el9.noarch.rpm 297 kB/s | 151 kB 00:00 2024-05-17T14:50:10.803 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:50:10.806 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:50:10.806 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-05-17T14:50:10.806 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:50:10.806 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-05-17T14:50:10.806 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 6.6 k 2024-05-17T14:50:10.806 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-05-17T14:50:10.806 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 2.4 M 2024-05-17T14:50:10.806 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 1.5 M 2024-05-17T14:50:10.806 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 274 k 2024-05-17T14:50:10.807 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 4.7 M 2024-05-17T14:50:10.807 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 17 M 2024-05-17T14:50:10.807 INFO:teuthology.orchestra.run.smithi134.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-17T14:50:10.807 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-17T14:50:10.807 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-17T14:50:10.807 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-17T14:50:10.807 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-17T14:50:10.807 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-17T14:50:10.807 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-17T14:50:10.807 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-17T14:50:10.807 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-17T14:50:10.807 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-17T14:50:10.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections 2024-05-17T14:50:10.808 INFO:teuthology.orchestra.run.smithi134.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-05-17T14:50:10.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-17T14:50:10.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-17T14:50:10.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-17T14:50:10.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-17T14:50:10.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-17T14:50:10.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-17T14:50:10.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-17T14:50:10.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-17T14:50:10.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-17T14:50:10.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-17T14:50:10.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-17T14:50:10.808 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-17T14:50:10.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-17T14:50:10.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-17T14:50:10.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-17T14:50:10.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-17T14:50:10.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-17T14:50:10.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-17T14:50:10.809 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-17T14:50:10.809 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:10.809 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-05-17T14:50:10.809 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:50:10.809 INFO:teuthology.orchestra.run.smithi134.stdout:Install 36 Packages 2024-05-17T14:50:10.809 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:10.810 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 30 M 2024-05-17T14:50:10.810 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 108 M 2024-05-17T14:50:10.810 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-05-17T14:50:10.979 INFO:teuthology.orchestra.run.smithi195.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 87 kB/s | 43 kB 00:00 2024-05-17T14:50:11.179 INFO:teuthology.orchestra.run.smithi195.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 53 kB/s | 11 kB 00:00 2024-05-17T14:50:11.213 INFO:teuthology.orchestra.run.smithi195.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 355 kB/s | 172 kB 00:00 2024-05-17T14:50:11.288 INFO:teuthology.orchestra.run.smithi195.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 671 kB/s | 358 kB 00:00 2024-05-17T14:50:11.430 INFO:teuthology.orchestra.run.smithi195.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 70 kB/s | 18 kB 00:00 2024-05-17T14:50:11.456 INFO:teuthology.orchestra.run.smithi195.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 95 kB/s | 23 kB 00:00 2024-05-17T14:50:11.531 INFO:teuthology.orchestra.run.smithi195.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 80 kB/s | 19 kB 00:00 2024-05-17T14:50:11.550 INFO:teuthology.orchestra.run.smithi134.stdout:(1/36): ceph-19.0.0-3735.gee59d86a.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-05-17T14:50:11.715 INFO:teuthology.orchestra.run.smithi195.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 69 kB/s | 20 kB 00:00 2024-05-17T14:50:11.773 INFO:teuthology.orchestra.run.smithi195.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 146 kB/s | 46 kB 00:00 2024-05-17T14:50:11.775 INFO:teuthology.orchestra.run.smithi134.stdout:(2/36): ceph-mds-19.0.0-3735.gee59d86a.el9.x86_ 6.1 MB/s | 2.4 MB 00:00 2024-05-17T14:50:11.857 INFO:teuthology.orchestra.run.smithi195.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 242 kB/s | 79 kB 00:00 2024-05-17T14:50:12.032 INFO:teuthology.orchestra.run.smithi195.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 63 kB/s | 16 kB 00:00 2024-05-17T14:50:12.174 INFO:teuthology.orchestra.run.smithi195.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 284 kB/s | 90 kB 00:00 2024-05-17T14:50:12.199 INFO:teuthology.orchestra.run.smithi195.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 562 kB/s | 272 kB 00:00 2024-05-17T14:50:12.283 INFO:teuthology.orchestra.run.smithi195.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 143 kB/s | 36 kB 00:00 2024-05-17T14:50:12.401 INFO:teuthology.orchestra.run.smithi134.stdout:(3/36): ceph-mon-19.0.0-3735.gee59d86a.el9.x86_ 5.5 MB/s | 4.7 MB 00:00 2024-05-17T14:50:12.442 INFO:teuthology.orchestra.run.smithi134.stdout:(4/36): ceph-mgr-19.0.0-3735.gee59d86a.el9.x86_ 1.4 MB/s | 1.5 MB 00:01 2024-05-17T14:50:12.484 INFO:teuthology.orchestra.run.smithi134.stdout:(5/36): ceph-mgr-modules-core-19.0.0-3735.gee59 3.2 MB/s | 274 kB 00:00 2024-05-17T14:50:12.633 INFO:teuthology.orchestra.run.smithi195.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 57 kB/s | 20 kB 00:00 2024-05-17T14:50:12.726 INFO:teuthology.orchestra.run.smithi134.stdout:(6/36): ceph-osd-19.0.0-3735.gee59d86a.el9.x86_ 18 MB/s | 17 MB 00:00 2024-05-17T14:50:12.875 INFO:teuthology.orchestra.run.smithi195.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 328 kB/s | 230 kB 00:00 2024-05-17T14:50:12.885 INFO:teuthology.orchestra.run.smithi134.stdout:(7/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 572 kB/s | 253 kB 00:00 2024-05-17T14:50:12.900 INFO:teuthology.orchestra.run.smithi195.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 610 kB/s | 427 kB 00:00 2024-05-17T14:50:12.903 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:50:12.904 INFO:teuthology.orchestra.run.smithi195.stdout:Total 5.2 MB/s | 30 MB 00:05 2024-05-17T14:50:12.927 INFO:teuthology.orchestra.run.smithi134.stdout:(8/36): python3-ply-3.11-14.el9.noarch.rpm 532 kB/s | 106 kB 00:00 2024-05-17T14:50:12.952 INFO:teuthology.orchestra.run.smithi134.stdout:(9/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-05-17T14:50:13.011 INFO:teuthology.orchestra.run.smithi134.stdout:(10/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-05-17T14:50:13.036 INFO:teuthology.orchestra.run.smithi134.stdout:(11/36): python3-requests-2.25.1-8.el9.noarch.r 1.1 MB/s | 125 kB 00:00 2024-05-17T14:50:13.048 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-17T14:50:13.088 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-17T14:50:13.088 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-17T14:50:13.212 INFO:teuthology.orchestra.run.smithi134.stdout:(12/36): python3-mako-1.1.4-6.el9.noarch.rpm 983 kB/s | 172 kB 00:00 2024-05-17T14:50:13.237 INFO:teuthology.orchestra.run.smithi134.stdout:(13/36): lua-5.4.4-4.el9.x86_64.rpm 834 kB/s | 188 kB 00:00 2024-05-17T14:50:13.270 INFO:teuthology.orchestra.run.smithi134.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 595 kB/s | 35 kB 00:00 2024-05-17T14:50:13.296 INFO:teuthology.orchestra.run.smithi134.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 873 kB/s | 51 kB 00:00 2024-05-17T14:50:13.330 INFO:teuthology.orchestra.run.smithi134.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 708 kB/s | 42 kB 00:00 2024-05-17T14:50:13.355 INFO:teuthology.orchestra.run.smithi134.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 379 kB/s | 22 kB 00:00 2024-05-17T14:50:13.397 INFO:teuthology.orchestra.run.smithi134.stdout:(18/36): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2024-05-17T14:50:13.551 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-17T14:50:13.552 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-17T14:50:13.555 INFO:teuthology.orchestra.run.smithi134.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 217 kB/s | 43 kB 00:00 2024-05-17T14:50:13.614 INFO:teuthology.orchestra.run.smithi134.stdout:(20/36): luarocks-3.9.2-5.el9.noarch.rpm 533 kB/s | 151 kB 00:00 2024-05-17T14:50:13.647 INFO:teuthology.orchestra.run.smithi134.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 685 kB/s | 172 kB 00:00 2024-05-17T14:50:13.673 INFO:teuthology.orchestra.run.smithi134.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 182 kB/s | 11 kB 00:00 2024-05-17T14:50:13.714 INFO:teuthology.orchestra.run.smithi134.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2024-05-17T14:50:13.740 INFO:teuthology.orchestra.run.smithi134.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-05-17T14:50:13.765 INFO:teuthology.orchestra.run.smithi134.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 251 kB/s | 23 kB 00:00 2024-05-17T14:50:13.790 INFO:teuthology.orchestra.run.smithi134.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-05-17T14:50:13.824 INFO:teuthology.orchestra.run.smithi134.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 336 kB/s | 20 kB 00:00 2024-05-17T14:50:13.849 INFO:teuthology.orchestra.run.smithi134.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 553 kB/s | 46 kB 00:00 2024-05-17T14:50:13.875 INFO:teuthology.orchestra.run.smithi134.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 939 kB/s | 79 kB 00:00 2024-05-17T14:50:13.908 INFO:teuthology.orchestra.run.smithi134.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-05-17T14:50:13.934 INFO:teuthology.orchestra.run.smithi134.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-05-17T14:50:13.959 INFO:teuthology.orchestra.run.smithi134.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-05-17T14:50:13.984 INFO:teuthology.orchestra.run.smithi134.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 473 kB/s | 36 kB 00:00 2024-05-17T14:50:14.009 INFO:teuthology.orchestra.run.smithi134.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-05-17T14:50:14.051 INFO:teuthology.orchestra.run.smithi134.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 299 kB/s | 20 kB 00:00 2024-05-17T14:50:14.076 INFO:teuthology.orchestra.run.smithi134.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-05-17T14:50:14.078 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:50:14.078 INFO:teuthology.orchestra.run.smithi134.stdout:Total 9.2 MB/s | 30 MB 00:03 2024-05-17T14:50:14.222 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-05-17T14:50:14.262 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-05-17T14:50:14.262 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-05-17T14:50:14.718 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-05-17T14:50:14.718 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-05-17T14:50:14.806 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-17T14:50:14.861 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-05-17T14:50:14.894 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-05-17T14:50:14.922 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-05-17T14:50:14.991 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-05-17T14:50:15.029 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/36 2024-05-17T14:50:15.061 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-05-17T14:50:15.152 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-05-17T14:50:15.223 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-05-17T14:50:15.264 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-05-17T14:50:15.299 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-05-17T14:50:15.329 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-05-17T14:50:15.364 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-05-17T14:50:15.397 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-05-17T14:50:15.437 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-05-17T14:50:15.481 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-05-17T14:50:15.515 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-05-17T14:50:15.553 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-05-17T14:50:15.626 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-05-17T14:50:15.765 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-05-17T14:50:15.844 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-05-17T14:50:15.898 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-05-17T14:50:15.937 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-05-17T14:50:15.983 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-05-17T14:50:15.985 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-05-17T14:50:16.040 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-05-17T14:50:16.054 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-05-17T14:50:16.087 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-05-17T14:50:16.115 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-05-17T14:50:16.186 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-05-17T14:50:16.227 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-05-17T14:50:16.231 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/36 2024-05-17T14:50:16.263 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-05-17T14:50:16.282 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-05-17T14:50:16.360 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-05-17T14:50:16.362 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-05-17T14:50:16.448 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-05-17T14:50:16.450 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-05-17T14:50:16.490 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-05-17T14:50:16.492 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-05-17T14:50:16.526 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-05-17T14:50:16.559 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-05-17T14:50:16.590 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-05-17T14:50:16.616 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-05-17T14:50:16.624 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-05-17T14:50:16.664 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-05-17T14:50:16.700 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-05-17T14:50:16.734 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-05-17T14:50:16.772 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-05-17T14:50:16.793 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3735.gee59d86a.el9. 31/36 2024-05-17T14:50:16.819 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-2:19.0.0-3735.gee59d86a.el9.x86_64 32/36 2024-05-17T14:50:16.845 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-05-17T14:50:16.846 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3735.gee59d86a.el9.x86_64 32/36 2024-05-17T14:50:16.846 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-17T14:50:16.847 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-17T14:50:16.847 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-17T14:50:16.847 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-17T14:50:16.847 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:16.984 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-05-17T14:50:17.063 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-05-17T14:50:17.125 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-05-17T14:50:17.164 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-05-17T14:50:17.212 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-05-17T14:50:17.267 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-05-17T14:50:17.453 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-05-17T14:50:17.526 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-05-17T14:50:17.596 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-05-17T14:50:17.684 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-05-17T14:50:17.725 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-05-17T14:50:17.851 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-05-17T14:50:18.013 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3735.gee59d86a.el9. 31/36 2024-05-17T14:50:18.044 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:19.0.0-3735.gee59d86a.el9.x86_64 32/36 2024-05-17T14:50:18.071 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3735.gee59d86a.el9.x86_64 32/36 2024-05-17T14:50:18.071 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-17T14:50:18.071 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-17T14:50:18.071 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-17T14:50:18.071 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-17T14:50:18.071 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:18.295 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-osd-2:19.0.0-3735.gee59d86a.el9.x86_64 33/36 2024-05-17T14:50:18.323 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-osd-2:19.0.0-3735.gee59d86a.el9.x86_64 33/36 2024-05-17T14:50:18.323 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-17T14:50:18.323 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-17T14:50:18.323 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-17T14:50:18.323 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-17T14:50:18.323 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:18.771 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mon-2:19.0.0-3735.gee59d86a.el9.x86_64 34/36 2024-05-17T14:50:18.795 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mon-2:19.0.0-3735.gee59d86a.el9.x86_64 34/36 2024-05-17T14:50:18.795 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-17T14:50:18.795 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-17T14:50:18.795 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-17T14:50:18.795 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-17T14:50:18.795 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:19.021 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mds-2:19.0.0-3735.gee59d86a.el9.x86_64 35/36 2024-05-17T14:50:19.045 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mds-2:19.0.0-3735.gee59d86a.el9.x86_64 35/36 2024-05-17T14:50:19.045 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-17T14:50:19.045 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-17T14:50:19.045 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-17T14:50:19.045 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-17T14:50:19.045 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:19.109 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-2:19.0.0-3735.gee59d86a.el9.x86_64 36/36 2024-05-17T14:50:19.520 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:19.0.0-3735.gee59d86a.el9.x86_64 33/36 2024-05-17T14:50:19.547 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:19.0.0-3735.gee59d86a.el9.x86_64 33/36 2024-05-17T14:50:19.548 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-17T14:50:19.548 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-17T14:50:19.548 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-17T14:50:19.548 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-17T14:50:19.548 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:19.996 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:19.0.0-3735.gee59d86a.el9.x86_64 34/36 2024-05-17T14:50:20.020 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:19.0.0-3735.gee59d86a.el9.x86_64 34/36 2024-05-17T14:50:20.020 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-17T14:50:20.020 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-17T14:50:20.020 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-17T14:50:20.020 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-17T14:50:20.020 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:20.247 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:19.0.0-3735.gee59d86a.el9.x86_64 35/36 2024-05-17T14:50:20.270 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:19.0.0-3735.gee59d86a.el9.x86_64 35/36 2024-05-17T14:50:20.271 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-17T14:50:20.271 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-17T14:50:20.271 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-17T14:50:20.271 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-17T14:50:20.271 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:20.335 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:19.0.0-3735.gee59d86a.el9.x86_64 36/36 2024-05-17T14:50:20.898 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-2:19.0.0-3735.gee59d86a.el9.x86_64 36/36 2024-05-17T14:50:20.899 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-2:19.0.0-3735.gee59d86a.el9.x86_64 1/36 2024-05-17T14:50:20.899 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mds-2:19.0.0-3735.gee59d86a.el9.x86_64 2/36 2024-05-17T14:50:20.899 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-2:19.0.0-3735.gee59d86a.el9.x86_64 3/36 2024-05-17T14:50:20.899 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mon-2:19.0.0-3735.gee59d86a.el9.x86_64 4/36 2024-05-17T14:50:20.899 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-osd-2:19.0.0-3735.gee59d86a.el9.x86_64 5/36 2024-05-17T14:50:20.899 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3735.gee59d86a.el9. 6/36 2024-05-17T14:50:20.899 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-05-17T14:50:20.899 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-05-17T14:50:20.899 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-05-17T14:50:20.899 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-05-17T14:50:20.899 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-05-17T14:50:20.899 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-05-17T14:50:20.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-05-17T14:50:20.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-05-17T14:50:20.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-05-17T14:50:20.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-05-17T14:50:20.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-05-17T14:50:20.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-05-17T14:50:20.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 19/36 2024-05-17T14:50:20.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-05-17T14:50:20.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-05-17T14:50:20.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-05-17T14:50:20.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-05-17T14:50:20.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-05-17T14:50:20.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-05-17T14:50:20.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-05-17T14:50:20.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-05-17T14:50:20.900 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-05-17T14:50:20.901 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-05-17T14:50:20.901 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-05-17T14:50:20.902 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-05-17T14:50:20.902 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-05-17T14:50:20.902 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-05-17T14:50:20.902 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-05-17T14:50:20.902 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-05-17T14:50:21.981 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-05-17T14:50:21.982 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:21.982 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-17T14:50:21.982 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:50:21.982 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:50:21.982 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:50:21.982 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:50:21.982 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:50:21.982 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:50:21.982 INFO:teuthology.orchestra.run.smithi195.stdout: lua-5.4.4-4.el9.x86_64 2024-05-17T14:50:21.982 INFO:teuthology.orchestra.run.smithi195.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-17T14:50:21.983 INFO:teuthology.orchestra.run.smithi195.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-17T14:50:21.983 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-17T14:50:21.983 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-17T14:50:21.983 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-17T14:50:21.983 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-17T14:50:21.983 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-17T14:50:21.983 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-17T14:50:21.983 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-17T14:50:21.983 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-17T14:50:21.983 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-17T14:50:21.983 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-17T14:50:21.983 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-17T14:50:21.983 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-17T14:50:21.984 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-17T14:50:21.984 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-17T14:50:21.984 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-17T14:50:21.984 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ply-3.11-14.el9.noarch 2024-05-17T14:50:21.984 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-17T14:50:21.984 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-17T14:50:21.984 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-17T14:50:21.984 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-17T14:50:21.984 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-17T14:50:21.984 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-17T14:50:21.984 INFO:teuthology.orchestra.run.smithi195.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-17T14:50:21.984 INFO:teuthology.orchestra.run.smithi195.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-17T14:50:21.984 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-17T14:50:21.985 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-17T14:50:21.985 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-17T14:50:21.985 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:21.985 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:50:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-2:19.0.0-3735.gee59d86a.el9.x86_64 36/36 2024-05-17T14:50:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:19.0.0-3735.gee59d86a.el9.x86_64 1/36 2024-05-17T14:50:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:19.0.0-3735.gee59d86a.el9.x86_64 2/36 2024-05-17T14:50:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:19.0.0-3735.gee59d86a.el9.x86_64 3/36 2024-05-17T14:50:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:19.0.0-3735.gee59d86a.el9.x86_64 4/36 2024-05-17T14:50:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:19.0.0-3735.gee59d86a.el9.x86_64 5/36 2024-05-17T14:50:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3735.gee59d86a.el9. 6/36 2024-05-17T14:50:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-05-17T14:50:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-05-17T14:50:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-05-17T14:50:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-05-17T14:50:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-05-17T14:50:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-05-17T14:50:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-05-17T14:50:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-05-17T14:50:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-05-17T14:50:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-05-17T14:50:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-05-17T14:50:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-05-17T14:50:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 19/36 2024-05-17T14:50:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-05-17T14:50:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-05-17T14:50:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-05-17T14:50:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-05-17T14:50:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-05-17T14:50:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-05-17T14:50:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-05-17T14:50:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-05-17T14:50:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-05-17T14:50:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-05-17T14:50:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-05-17T14:50:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-05-17T14:50:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-05-17T14:50:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-05-17T14:50:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-05-17T14:50:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-05-17T14:50:22.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-base 2024-05-17T14:50:22.899 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:56 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:50:22.980 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-base-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:50:23.014 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-05-17T14:50:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:23.015 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-05-17T14:50:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:50:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:50:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:50:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:50:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:50:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:50:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: lua-5.4.4-4.el9.x86_64 2024-05-17T14:50:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-17T14:50:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-17T14:50:23.015 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-17T14:50:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-17T14:50:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-17T14:50:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-17T14:50:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-17T14:50:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-17T14:50:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-17T14:50:23.016 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-17T14:50:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-17T14:50:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-17T14:50:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-17T14:50:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-17T14:50:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-17T14:50:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-17T14:50:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-17T14:50:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply-3.11-14.el9.noarch 2024-05-17T14:50:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-17T14:50:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-17T14:50:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-17T14:50:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-17T14:50:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-17T14:50:23.017 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-17T14:50:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-17T14:50:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-17T14:50:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-17T14:50:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-17T14:50:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-17T14:50:23.018 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:23.018 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:50:23.021 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:50:23.022 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-17T14:50:23.022 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:50:23.087 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install cephadm 2024-05-17T14:50:23.410 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base 2024-05-17T14:50:23.653 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:57 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:50:23.775 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:50:23.776 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:50:23.776 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-17T14:50:23.776 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:50:23.776 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-17T14:50:23.776 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 763 k 2024-05-17T14:50:23.776 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:23.776 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-17T14:50:23.776 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:50:23.776 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-05-17T14:50:23.776 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:23.777 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 763 k 2024-05-17T14:50:23.777 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 770 k 2024-05-17T14:50:23.777 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-17T14:50:23.987 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:54 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:50:24.068 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:50:24.079 INFO:teuthology.orchestra.run.smithi195.stdout:cephadm-19.0.0-3735.gee59d86a.el9.noarch.rpm 2.5 MB/s | 763 kB 00:00 2024-05-17T14:50:24.080 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:50:24.080 INFO:teuthology.orchestra.run.smithi195.stdout:Total 2.5 MB/s | 763 kB 00:00 2024-05-17T14:50:24.080 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-17T14:50:24.085 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-17T14:50:24.085 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-17T14:50:24.091 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-17T14:50:24.091 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-17T14:50:24.109 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:50:24.110 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-05-17T14:50:24.110 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:50:24.160 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-17T14:50:24.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install cephadm 2024-05-17T14:50:24.541 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 1/1 2024-05-17T14:50:24.625 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 1/1 2024-05-17T14:50:24.749 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:55 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:50:24.873 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:50:24.873 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:50:24.873 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-05-17T14:50:24.873 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:50:24.873 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-05-17T14:50:24.873 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 763 k 2024-05-17T14:50:24.873 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:24.873 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-05-17T14:50:24.874 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:50:24.874 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-05-17T14:50:24.874 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:24.874 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 763 k 2024-05-17T14:50:24.874 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 770 k 2024-05-17T14:50:24.874 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-05-17T14:50:24.966 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 1/1 2024-05-17T14:50:25.146 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm-19.0.0-3735.gee59d86a.el9.noarch.rpm 2.7 MB/s | 763 kB 00:00 2024-05-17T14:50:25.147 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:50:25.147 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.7 MB/s | 763 kB 00:00 2024-05-17T14:50:25.147 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-05-17T14:50:25.152 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-05-17T14:50:25.153 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-05-17T14:50:25.158 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-05-17T14:50:25.159 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-05-17T14:50:25.226 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 1/1 2024-05-17T14:50:25.226 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:25.226 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-17T14:50:25.227 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:50:25.227 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:25.227 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:50:25.232 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-05-17T14:50:25.436 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-immutable-object-cache 2024-05-17T14:50:25.634 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 1/1 2024-05-17T14:50:25.742 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 1/1 2024-05-17T14:50:26.020 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:00 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:50:26.104 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 1/1 2024-05-17T14:50:26.146 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:50:26.146 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:50:26.147 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-05-17T14:50:26.147 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:50:26.147 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-17T14:50:26.147 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 145 k 2024-05-17T14:50:26.147 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:26.147 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-17T14:50:26.147 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:50:26.147 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-05-17T14:50:26.147 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:26.148 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 145 k 2024-05-17T14:50:26.148 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 435 k 2024-05-17T14:50:26.148 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-17T14:50:26.357 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 1/1 2024-05-17T14:50:26.357 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:26.357 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-05-17T14:50:26.357 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:50:26.357 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:26.357 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:50:26.386 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-immutable-object-cache-19.0.0-3735.gee59d8 610 kB/s | 145 kB 00:00 2024-05-17T14:50:26.386 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:50:26.387 INFO:teuthology.orchestra.run.smithi195.stdout:Total 605 kB/s | 145 kB 00:00 2024-05-17T14:50:26.387 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-17T14:50:26.394 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-17T14:50:26.394 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-17T14:50:26.455 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-17T14:50:26.455 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-17T14:50:26.568 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache 2024-05-17T14:50:26.601 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-17T14:50:26.687 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3735.gee59d86a. 1/1 2024-05-17T14:50:26.713 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3735.gee59d86a. 1/1 2024-05-17T14:50:26.713 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-17T14:50:26.713 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-17T14:50:26.713 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:27.154 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:58 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:50:27.281 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:50:27.282 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:50:27.282 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-05-17T14:50:27.282 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:50:27.282 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-05-17T14:50:27.282 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 145 k 2024-05-17T14:50:27.282 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:27.283 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-05-17T14:50:27.283 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:50:27.283 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-05-17T14:50:27.283 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:27.283 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 145 k 2024-05-17T14:50:27.283 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 435 k 2024-05-17T14:50:27.283 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-05-17T14:50:27.506 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-19.0.0-3735.gee59d8 656 kB/s | 145 kB 00:00 2024-05-17T14:50:27.507 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:50:27.507 INFO:teuthology.orchestra.run.smithi134.stdout:Total 652 kB/s | 145 kB 00:00 2024-05-17T14:50:27.507 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-05-17T14:50:27.512 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-05-17T14:50:27.512 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-05-17T14:50:27.573 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-05-17T14:50:27.573 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-05-17T14:50:27.583 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3735.gee59d86a. 1/1 2024-05-17T14:50:27.583 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:27.583 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-17T14:50:27.583 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:50:27.583 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:27.583 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:50:27.730 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-05-17T14:50:27.797 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3735.gee59d86a. 1/1 2024-05-17T14:50:27.805 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr 2024-05-17T14:50:27.822 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3735.gee59d86a. 1/1 2024-05-17T14:50:27.822 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-17T14:50:27.822 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-17T14:50:27.822 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:28.388 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:02 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:50:28.468 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-mgr-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:50:28.511 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:50:28.511 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-17T14:50:28.512 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:50:28.575 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-dashboard 2024-05-17T14:50:28.724 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3735.gee59d86a. 1/1 2024-05-17T14:50:28.725 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:28.725 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-05-17T14:50:28.725 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:50:28.725 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:28.725 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:50:28.951 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr 2024-05-17T14:50:29.142 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:03 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:50:29.266 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:50:29.267 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:50:29.268 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-17T14:50:29.268 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:50:29.268 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-17T14:50:29.268 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 3.5 M 2024-05-17T14:50:29.268 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-17T14:50:29.268 INFO:teuthology.orchestra.run.smithi195.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-17T14:50:29.268 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 34 k 2024-05-17T14:50:29.268 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 16 k 2024-05-17T14:50:29.268 INFO:teuthology.orchestra.run.smithi195.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-17T14:50:29.268 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-17T14:50:29.269 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-17T14:50:29.269 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-17T14:50:29.269 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-17T14:50:29.269 INFO:teuthology.orchestra.run.smithi195.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-17T14:50:29.269 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-17T14:50:29.269 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-17T14:50:29.269 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:29.269 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-17T14:50:29.269 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:50:29.269 INFO:teuthology.orchestra.run.smithi195.stdout:Install 12 Packages 2024-05-17T14:50:29.270 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:29.270 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 8.7 M 2024-05-17T14:50:29.270 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 104 M 2024-05-17T14:50:29.270 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-17T14:50:29.534 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:00 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:50:29.618 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-mgr-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:50:29.662 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:50:29.662 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-05-17T14:50:29.663 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:50:29.734 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-dashboard 2024-05-17T14:50:29.790 INFO:teuthology.orchestra.run.smithi195.stdout:(1/12): ceph-grafana-dashboards-19.0.0-3735.gee 158 kB/s | 34 kB 00:00 2024-05-17T14:50:29.890 INFO:teuthology.orchestra.run.smithi195.stdout:(2/12): ceph-prometheus-alerts-19.0.0-3735.gee5 52 kB/s | 16 kB 00:00 2024-05-17T14:50:30.049 INFO:teuthology.orchestra.run.smithi195.stdout:(3/12): ceph-mgr-dashboard-19.0.0-3735.gee59d86 7.4 MB/s | 3.5 MB 00:00 2024-05-17T14:50:30.282 INFO:teuthology.orchestra.run.smithi195.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.6 MB/s | 863 kB 00:00 2024-05-17T14:50:30.303 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:01 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:50:30.399 INFO:teuthology.orchestra.run.smithi195.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 529 kB/s | 269 kB 00:00 2024-05-17T14:50:30.428 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:50:30.429 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:50:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-05-17T14:50:30.429 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:50:30.429 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-05-17T14:50:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 3.5 M 2024-05-17T14:50:30.429 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-05-17T14:50:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-17T14:50:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 34 k 2024-05-17T14:50:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 16 k 2024-05-17T14:50:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-17T14:50:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-17T14:50:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-17T14:50:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-17T14:50:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-17T14:50:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-17T14:50:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-17T14:50:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-17T14:50:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:30.430 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-05-17T14:50:30.430 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:50:30.430 INFO:teuthology.orchestra.run.smithi134.stdout:Install 12 Packages 2024-05-17T14:50:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:30.431 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 8.7 M 2024-05-17T14:50:30.431 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 104 M 2024-05-17T14:50:30.431 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-05-17T14:50:30.641 INFO:teuthology.orchestra.run.smithi195.stdout:(6/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.2 MB/s | 1.0 MB 00:00 2024-05-17T14:50:30.666 INFO:teuthology.orchestra.run.smithi195.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 73 kB/s | 19 kB 00:00 2024-05-17T14:50:30.842 INFO:teuthology.orchestra.run.smithi134.stdout:(1/12): ceph-prometheus-alerts-19.0.0-3735.gee5 85 kB/s | 16 kB 00:00 2024-05-17T14:50:30.925 INFO:teuthology.orchestra.run.smithi195.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 853 kB/s | 548 kB 00:00 2024-05-17T14:50:30.942 INFO:teuthology.orchestra.run.smithi134.stdout:(2/12): ceph-grafana-dashboards-19.0.0-3735.gee 116 kB/s | 34 kB 00:00 2024-05-17T14:50:31.134 INFO:teuthology.orchestra.run.smithi195.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 309 kB/s | 144 kB 00:00 2024-05-17T14:50:31.176 INFO:teuthology.orchestra.run.smithi195.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 123 kB/s | 31 kB 00:00 2024-05-17T14:50:31.192 INFO:teuthology.orchestra.run.smithi134.stdout:(3/12): ceph-mgr-dashboard-19.0.0-3735.gee59d86 6.5 MB/s | 3.5 MB 00:00 2024-05-17T14:50:31.484 INFO:teuthology.orchestra.run.smithi134.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.2 MB/s | 863 kB 00:00 2024-05-17T14:50:31.626 INFO:teuthology.orchestra.run.smithi195.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 383 kB/s | 188 kB 00:00 2024-05-17T14:50:31.793 INFO:teuthology.orchestra.run.smithi134.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 317 kB/s | 269 kB 00:00 2024-05-17T14:50:31.835 INFO:teuthology.orchestra.run.smithi134.stdout:(6/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:00 2024-05-17T14:50:31.968 INFO:teuthology.orchestra.run.smithi134.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-05-17T14:50:32.060 INFO:teuthology.orchestra.run.smithi134.stdout:(8/12): grpc-data-1.46.7-10.el9.noarch.rpm 73 kB/s | 19 kB 00:00 2024-05-17T14:50:32.085 INFO:teuthology.orchestra.run.smithi134.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-05-17T14:50:32.134 INFO:teuthology.orchestra.run.smithi195.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 1.4 MB/s | 2.0 MB 00:01 2024-05-17T14:50:32.137 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:50:32.138 INFO:teuthology.orchestra.run.smithi195.stdout:Total 3.0 MB/s | 8.7 MB 00:02 2024-05-17T14:50:32.191 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-17T14:50:32.194 INFO:teuthology.orchestra.run.smithi134.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-05-17T14:50:32.208 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-17T14:50:32.209 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-17T14:50:32.388 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-17T14:50:32.389 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-17T14:50:32.985 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-17T14:50:33.060 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-05-17T14:50:33.113 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-05-17T14:50:33.167 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-05-17T14:50:33.197 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-05-17T14:50:33.296 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-05-17T14:50:33.426 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-05-17T14:50:33.526 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-05-17T14:50:33.570 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-05-17T14:50:33.597 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-05-17T14:50:33.629 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3735.gee59d86a.el9 10/12 2024-05-17T14:50:34.681 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3735.gee59d86a.el 11/12 2024-05-17T14:50:34.721 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3735.gee59d86a.el9.noa 12/12 2024-05-17T14:50:35.103 INFO:teuthology.orchestra.run.smithi134.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 10 kB/s | 31 kB 00:03 2024-05-17T14:50:35.603 INFO:teuthology.orchestra.run.smithi134.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 554 kB/s | 2.0 MB 00:03 2024-05-17T14:50:35.608 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:50:35.608 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.7 MB/s | 8.7 MB 00:05 2024-05-17T14:50:35.661 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-05-17T14:50:35.680 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-05-17T14:50:35.680 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-05-17T14:50:35.861 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-05-17T14:50:35.861 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-05-17T14:50:35.980 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3735.gee59d86a.el9.noa 12/12 2024-05-17T14:50:35.980 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3735.gee59d86a.el 1/12 2024-05-17T14:50:35.980 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3735.gee59d86a.el9.noa 2/12 2024-05-17T14:50:35.980 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3735.gee59d86a.el9 3/12 2024-05-17T14:50:35.980 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-05-17T14:50:35.980 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-05-17T14:50:35.980 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-05-17T14:50:35.980 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-05-17T14:50:35.981 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-05-17T14:50:35.981 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-05-17T14:50:35.981 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-05-17T14:50:35.981 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-05-17T14:50:36.283 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-05-17T14:50:36.283 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:36.283 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-17T14:50:36.283 INFO:teuthology.orchestra.run.smithi195.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-17T14:50:36.283 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:50:36.283 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:50:36.283 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:50:36.283 INFO:teuthology.orchestra.run.smithi195.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-17T14:50:36.284 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-17T14:50:36.284 INFO:teuthology.orchestra.run.smithi195.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-17T14:50:36.284 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-17T14:50:36.284 INFO:teuthology.orchestra.run.smithi195.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-17T14:50:36.284 INFO:teuthology.orchestra.run.smithi195.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-17T14:50:36.284 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-17T14:50:36.284 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-17T14:50:36.284 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:36.284 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:50:36.471 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-05-17T14:50:36.530 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-17T14:50:36.545 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-05-17T14:50:36.589 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-05-17T14:50:36.635 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-05-17T14:50:36.674 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-05-17T14:50:36.773 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-05-17T14:50:36.909 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-05-17T14:50:37.003 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-05-17T14:50:37.046 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-05-17T14:50:37.074 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-05-17T14:50:37.107 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3735.gee59d86a.el9 10/12 2024-05-17T14:50:37.109 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:11 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:50:37.241 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:50:37.243 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================= 2024-05-17T14:50:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-17T14:50:37.243 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================= 2024-05-17T14:50:37.243 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-17T14:50:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 7.4 M 2024-05-17T14:50:37.243 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-17T14:50:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-17T14:50:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-17T14:50:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-17T14:50:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-17T14:50:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-17T14:50:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-17T14:50:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-17T14:50:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-17T14:50:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-17T14:50:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-17T14:50:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-17T14:50:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:37.244 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-17T14:50:37.244 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================= 2024-05-17T14:50:37.244 INFO:teuthology.orchestra.run.smithi195.stdout:Install 12 Packages 2024-05-17T14:50:37.245 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:50:37.245 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 38 M 2024-05-17T14:50:37.245 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 209 M 2024-05-17T14:50:37.245 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-17T14:50:37.835 INFO:teuthology.orchestra.run.smithi195.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-05-17T14:50:37.885 INFO:teuthology.orchestra.run.smithi195.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 803 kB 00:00 2024-05-17T14:50:38.094 INFO:teuthology.orchestra.run.smithi195.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 126 kB/s | 33 kB 00:00 2024-05-17T14:50:38.155 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3735.gee59d86a.el 11/12 2024-05-17T14:50:38.211 INFO:teuthology.orchestra.run.smithi195.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 156 kB/s | 18 kB 00:00 2024-05-17T14:50:38.215 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3735.gee59d86a.el9.noa 12/12 2024-05-17T14:50:38.453 INFO:teuthology.orchestra.run.smithi195.stdout:(5/12): ceph-mgr-diskprediction-local-19.0.0-37 9.7 MB/s | 7.4 MB 00:00 2024-05-17T14:50:38.478 INFO:teuthology.orchestra.run.smithi195.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 132 kB/s | 35 kB 00:00 2024-05-17T14:50:39.053 INFO:teuthology.orchestra.run.smithi195.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-05-17T14:50:39.078 INFO:teuthology.orchestra.run.smithi195.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 407 kB/s | 244 kB 00:00 2024-05-17T14:50:39.262 INFO:teuthology.orchestra.run.smithi195.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.3 MB/s | 250 kB 00:00 2024-05-17T14:50:39.553 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3735.gee59d86a.el9.noa 12/12 2024-05-17T14:50:39.553 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3735.gee59d86a.el 1/12 2024-05-17T14:50:39.553 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3735.gee59d86a.el9.noa 2/12 2024-05-17T14:50:39.553 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3735.gee59d86a.el9 3/12 2024-05-17T14:50:39.553 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-05-17T14:50:39.554 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-05-17T14:50:39.554 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-05-17T14:50:39.554 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-05-17T14:50:39.554 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-05-17T14:50:39.554 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-05-17T14:50:39.554 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-05-17T14:50:39.554 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-05-17T14:50:39.879 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-05-17T14:50:39.879 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:39.879 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-05-17T14:50:39.879 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-17T14:50:39.879 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:50:39.879 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:50:39.879 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:50:39.880 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-17T14:50:39.880 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-17T14:50:39.880 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-17T14:50:39.880 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-17T14:50:39.880 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-17T14:50:39.880 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-17T14:50:39.880 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-17T14:50:39.880 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-17T14:50:39.880 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:39.880 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:50:40.142 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-17T14:50:40.729 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:11 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:50:40.867 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:50:40.868 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-05-17T14:50:40.868 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-05-17T14:50:40.868 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-05-17T14:50:40.868 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-05-17T14:50:40.868 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 7.4 M 2024-05-17T14:50:40.868 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-05-17T14:50:40.869 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-17T14:50:40.869 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-17T14:50:40.869 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-17T14:50:40.869 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-17T14:50:40.869 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-17T14:50:40.869 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-17T14:50:40.869 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-17T14:50:40.869 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-17T14:50:40.869 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-17T14:50:40.869 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-17T14:50:40.869 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-17T14:50:40.869 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:40.870 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-05-17T14:50:40.870 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-05-17T14:50:40.870 INFO:teuthology.orchestra.run.smithi134.stdout:Install 12 Packages 2024-05-17T14:50:40.870 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:40.870 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 38 M 2024-05-17T14:50:40.870 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 209 M 2024-05-17T14:50:40.871 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-05-17T14:50:40.912 INFO:teuthology.orchestra.run.smithi195.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 1.9 MB/s | 4.7 MB 00:02 2024-05-17T14:50:41.154 INFO:teuthology.orchestra.run.smithi134.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 195 kB 00:00 2024-05-17T14:50:41.204 INFO:teuthology.orchestra.run.smithi134.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.7 MB/s | 803 kB 00:00 2024-05-17T14:50:41.413 INFO:teuthology.orchestra.run.smithi134.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 127 kB/s | 33 kB 00:00 2024-05-17T14:50:41.638 INFO:teuthology.orchestra.run.smithi134.stdout:(4/12): ceph-mgr-diskprediction-local-19.0.0-37 12 MB/s | 7.4 MB 00:00 2024-05-17T14:50:41.680 INFO:teuthology.orchestra.run.smithi134.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 68 kB/s | 18 kB 00:00 2024-05-17T14:50:41.906 INFO:teuthology.orchestra.run.smithi134.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 132 kB/s | 35 kB 00:00 2024-05-17T14:50:42.656 INFO:teuthology.orchestra.run.smithi134.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 325 kB/s | 244 kB 00:00 2024-05-17T14:50:43.520 INFO:teuthology.orchestra.run.smithi195.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.1 MB/s | 5.1 MB 00:04 2024-05-17T14:50:44.348 INFO:teuthology.orchestra.run.smithi134.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 974 kB/s | 3.0 MB 00:03 2024-05-17T14:50:44.873 INFO:teuthology.orchestra.run.smithi134.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 477 kB/s | 250 kB 00:00 2024-05-17T14:50:45.498 INFO:teuthology.orchestra.run.smithi134.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 1.2 MB/s | 4.7 MB 00:03 2024-05-17T14:50:46.565 INFO:teuthology.orchestra.run.smithi134.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.3 MB/s | 5.1 MB 00:03 2024-05-17T14:50:50.623 INFO:teuthology.orchestra.run.smithi134.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.8 MB/s | 16 MB 00:05 2024-05-17T14:50:50.624 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:50:50.625 INFO:teuthology.orchestra.run.smithi134.stdout:Total 3.9 MB/s | 38 MB 00:09 2024-05-17T14:50:50.986 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-05-17T14:50:51.011 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-05-17T14:50:51.011 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-05-17T14:50:51.452 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-05-17T14:50:51.453 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-05-17T14:50:52.203 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-05-17T14:50:52.279 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-17T14:50:52.321 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-17T14:50:52.421 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-17T14:50:52.472 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-17T14:50:52.650 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-17T14:50:52.680 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-17T14:50:52.758 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-17T14:50:53.111 INFO:teuthology.orchestra.run.smithi195.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 1.2 MB/s | 16 MB 00:13 2024-05-17T14:50:53.112 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:50:53.113 INFO:teuthology.orchestra.run.smithi195.stdout:Total 2.4 MB/s | 38 MB 00:15 2024-05-17T14:50:53.401 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-17T14:50:53.472 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-17T14:50:53.499 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-17T14:50:53.499 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-17T14:50:53.518 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-17T14:50:53.933 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-17T14:50:53.934 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-17T14:50:54.689 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-17T14:50:54.756 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-17T14:50:54.761 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-17T14:50:54.801 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-17T14:50:54.900 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-17T14:50:54.952 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-17T14:50:55.130 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-17T14:50:55.168 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-17T14:50:55.245 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-17T14:50:55.839 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-17T14:50:55.872 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3735.gee59d 12/12 2024-05-17T14:50:55.892 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-17T14:50:56.015 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-17T14:50:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-17T14:50:58.259 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3735.gee59d 12/12 2024-05-17T14:50:58.259 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3735.gee59d 1/12 2024-05-17T14:50:58.259 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-17T14:50:58.259 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-17T14:50:58.259 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-17T14:50:58.259 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-17T14:50:58.259 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-17T14:50:58.259 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-17T14:50:58.259 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-17T14:50:58.259 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-17T14:50:58.259 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-17T14:50:58.260 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-17T14:50:58.326 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-17T14:50:58.359 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3735.gee59d 12/12 2024-05-17T14:50:58.663 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-17T14:50:58.663 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:58.663 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-05-17T14:50:58.663 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:50:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-17T14:50:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-17T14:50:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-17T14:50:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-17T14:50:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-17T14:50:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-17T14:50:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-17T14:50:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-17T14:50:58.665 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-17T14:50:58.665 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-17T14:50:58.665 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-17T14:50:58.665 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:58.665 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:50:59.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-rook 2024-05-17T14:50:59.614 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:30 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:50:59.738 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:50:59.739 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:50:59.739 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-05-17T14:50:59.739 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:50:59.739 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-05-17T14:50:59.739 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 49 k 2024-05-17T14:50:59.740 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-05-17T14:50:59.740 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-17T14:50:59.740 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-17T14:50:59.740 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-17T14:50:59.740 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-17T14:50:59.740 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-17T14:50:59.741 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-17T14:50:59.741 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-17T14:50:59.741 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-17T14:50:59.741 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-17T14:50:59.741 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-17T14:50:59.741 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-17T14:50:59.741 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-17T14:50:59.742 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:59.742 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-05-17T14:50:59.742 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:50:59.742 INFO:teuthology.orchestra.run.smithi134.stdout:Install 13 Packages 2024-05-17T14:50:59.742 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:50:59.743 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 2.2 M 2024-05-17T14:50:59.744 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 26 M 2024-05-17T14:50:59.744 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-05-17T14:51:00.091 INFO:teuthology.orchestra.run.smithi134.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 329 kB/s | 26 kB 00:00 2024-05-17T14:51:00.107 INFO:teuthology.orchestra.run.smithi134.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 202 kB/s | 19 kB 00:00 2024-05-17T14:51:00.174 INFO:teuthology.orchestra.run.smithi134.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.4 MB/s | 159 kB 00:00 2024-05-17T14:51:00.200 INFO:teuthology.orchestra.run.smithi134.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-05-17T14:51:00.225 INFO:teuthology.orchestra.run.smithi134.stdout:(5/13): ceph-mgr-rook-19.0.0-3735.gee59d86a.el9 232 kB/s | 49 kB 00:00 2024-05-17T14:51:00.276 INFO:teuthology.orchestra.run.smithi134.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 711 kB/s | 54 kB 00:00 2024-05-17T14:51:00.326 INFO:teuthology.orchestra.run.smithi134.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.8 MB/s | 279 kB 00:00 2024-05-17T14:51:00.518 INFO:teuthology.orchestra.run.smithi134.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2024-05-17T14:51:00.543 INFO:teuthology.orchestra.run.smithi134.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 62 kB/s | 15 kB 00:00 2024-05-17T14:51:00.693 INFO:teuthology.orchestra.run.smithi134.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 394 kB/s | 59 kB 00:00 2024-05-17T14:51:00.718 INFO:teuthology.orchestra.run.smithi134.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 524 kB/s | 206 kB 00:00 2024-05-17T14:51:00.810 INFO:teuthology.orchestra.run.smithi134.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 768 kB/s | 90 kB 00:00 2024-05-17T14:51:00.835 INFO:teuthology.orchestra.run.smithi134.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.2 MB/s | 1.0 MB 00:00 2024-05-17T14:51:00.838 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:00.838 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.0 MB/s | 2.2 MB 00:01 2024-05-17T14:51:00.902 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-05-17T14:51:00.917 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-05-17T14:51:00.917 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-05-17T14:51:00.919 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3735.gee59d 12/12 2024-05-17T14:51:00.919 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3735.gee59d 1/12 2024-05-17T14:51:00.919 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-17T14:51:00.919 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-17T14:51:00.919 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-17T14:51:00.919 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-17T14:51:00.919 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-17T14:51:00.919 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-17T14:51:00.919 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-17T14:51:00.920 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-17T14:51:00.920 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-17T14:51:00.920 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-17T14:51:01.003 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-05-17T14:51:01.004 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-05-17T14:51:01.299 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-17T14:51:01.299 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:01.299 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-17T14:51:01.299 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:51:01.299 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-17T14:51:01.299 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-17T14:51:01.299 INFO:teuthology.orchestra.run.smithi195.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-17T14:51:01.299 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-17T14:51:01.299 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-17T14:51:01.299 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-17T14:51:01.299 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-17T14:51:01.300 INFO:teuthology.orchestra.run.smithi195.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-17T14:51:01.300 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-17T14:51:01.300 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-17T14:51:01.300 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-17T14:51:01.300 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:01.300 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:01.509 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-05-17T14:51:01.655 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-rook 2024-05-17T14:51:01.675 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-17T14:51:01.728 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-17T14:51:01.792 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-17T14:51:01.829 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-17T14:51:01.870 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-17T14:51:01.964 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-17T14:51:02.074 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-17T14:51:02.130 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-17T14:51:02.232 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:36 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:51:02.355 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:51:02.356 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:02.356 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-17T14:51:02.356 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:02.356 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-17T14:51:02.356 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 49 k 2024-05-17T14:51:02.357 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-17T14:51:02.357 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-17T14:51:02.357 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-17T14:51:02.357 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-17T14:51:02.357 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-17T14:51:02.357 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-17T14:51:02.357 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-17T14:51:02.357 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-17T14:51:02.357 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-17T14:51:02.357 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-17T14:51:02.357 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-17T14:51:02.357 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-17T14:51:02.357 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-17T14:51:02.358 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:02.358 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-17T14:51:02.358 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:02.358 INFO:teuthology.orchestra.run.smithi195.stdout:Install 13 Packages 2024-05-17T14:51:02.358 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:02.358 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 2.2 M 2024-05-17T14:51:02.358 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 26 M 2024-05-17T14:51:02.359 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-17T14:51:02.621 INFO:teuthology.orchestra.run.smithi195.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 270 kB/s | 26 kB 00:00 2024-05-17T14:51:02.645 INFO:teuthology.orchestra.run.smithi195.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 160 kB/s | 19 kB 00:00 2024-05-17T14:51:02.813 INFO:teuthology.orchestra.run.smithi195.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.1 MB/s | 222 kB 00:00 2024-05-17T14:51:02.838 INFO:teuthology.orchestra.run.smithi195.stdout:(4/13): ceph-mgr-rook-19.0.0-3735.gee59d86a.el9 158 kB/s | 49 kB 00:00 2024-05-17T14:51:02.853 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-17T14:51:02.855 INFO:teuthology.orchestra.run.smithi195.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 762 kB/s | 159 kB 00:00 2024-05-17T14:51:02.911 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-17T14:51:02.944 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-17T14:51:02.992 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-17T14:51:03.022 INFO:teuthology.orchestra.run.smithi195.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 193 kB/s | 32 kB 00:00 2024-05-17T14:51:03.022 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:19.0.0-3735.gee59d86a.el9.noarch 13/13 2024-05-17T14:51:03.047 INFO:teuthology.orchestra.run.smithi195.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 257 kB/s | 54 kB 00:00 2024-05-17T14:51:03.097 INFO:teuthology.orchestra.run.smithi195.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 199 kB/s | 15 kB 00:00 2024-05-17T14:51:03.172 INFO:teuthology.orchestra.run.smithi195.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 775 kB/s | 279 kB 00:00 2024-05-17T14:51:03.364 INFO:teuthology.orchestra.run.smithi195.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 648 kB/s | 206 kB 00:00 2024-05-17T14:51:03.389 INFO:teuthology.orchestra.run.smithi195.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 272 kB/s | 59 kB 00:00 2024-05-17T14:51:03.423 INFO:teuthology.orchestra.run.smithi195.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-05-17T14:51:03.548 INFO:teuthology.orchestra.run.smithi195.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.3 MB/s | 1.0 MB 00:00 2024-05-17T14:51:03.549 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:03.549 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.8 MB/s | 2.2 MB 00:01 2024-05-17T14:51:03.613 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-17T14:51:03.627 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-17T14:51:03.627 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-17T14:51:03.711 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-17T14:51:03.712 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-17T14:51:03.781 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3735.gee59d86a.el9.noarch 13/13 2024-05-17T14:51:03.781 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3735.gee59d86a.el9.noarch 1/13 2024-05-17T14:51:03.781 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-17T14:51:03.781 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-17T14:51:03.781 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-17T14:51:03.781 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-17T14:51:03.781 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-17T14:51:03.781 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-17T14:51:03.781 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-17T14:51:03.781 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-17T14:51:03.781 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-17T14:51:03.782 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-17T14:51:03.782 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-17T14:51:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-17T14:51:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:04.109 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-05-17T14:51:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:51:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-17T14:51:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-17T14:51:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-17T14:51:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-17T14:51:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-17T14:51:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-17T14:51:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-17T14:51:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-17T14:51:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-17T14:51:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-17T14:51:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-17T14:51:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-17T14:51:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:04.110 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:51:04.220 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-17T14:51:04.358 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-cephadm 2024-05-17T14:51:04.403 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-17T14:51:04.466 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-17T14:51:04.521 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-17T14:51:04.559 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-17T14:51:04.600 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-17T14:51:04.694 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-17T14:51:04.804 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-17T14:51:04.852 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-17T14:51:04.935 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:35 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:51:05.058 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:51:05.058 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:05.059 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-05-17T14:51:05.059 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:05.059 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-05-17T14:51:05.059 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 143 k 2024-05-17T14:51:05.059 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-05-17T14:51:05.059 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-17T14:51:05.059 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-17T14:51:05.059 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-17T14:51:05.059 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-17T14:51:05.059 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-17T14:51:05.059 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:05.059 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-05-17T14:51:05.059 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:05.060 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-05-17T14:51:05.060 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:05.060 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 7.0 M 2024-05-17T14:51:05.060 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 33 M 2024-05-17T14:51:05.060 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-05-17T14:51:05.520 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): ceph-mgr-cephadm-19.0.0-3735.gee59d86a.e 401 kB/s | 143 kB 00:00 2024-05-17T14:51:05.568 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-17T14:51:05.624 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-17T14:51:05.658 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-17T14:51:05.707 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-17T14:51:05.736 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-rook-2:19.0.0-3735.gee59d86a.el9.noarch 13/13 2024-05-17T14:51:06.037 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 284 kB/s | 248 kB 00:00 2024-05-17T14:51:06.470 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.6 MB/s | 6.0 MB 00:01 2024-05-17T14:51:06.510 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3735.gee59d86a.el9.noarch 13/13 2024-05-17T14:51:06.510 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3735.gee59d86a.el9.noarch 1/13 2024-05-17T14:51:06.510 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-17T14:51:06.510 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-17T14:51:06.510 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-17T14:51:06.510 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-17T14:51:06.510 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-17T14:51:06.510 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-17T14:51:06.510 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-17T14:51:06.510 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-17T14:51:06.511 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-17T14:51:06.511 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-17T14:51:06.511 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-17T14:51:06.629 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 97 kB/s | 58 kB 00:00 2024-05-17T14:51:06.819 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-17T14:51:06.819 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:06.819 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-17T14:51:06.819 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:51:06.819 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-17T14:51:06.819 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-17T14:51:06.819 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-17T14:51:06.819 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-17T14:51:06.820 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-17T14:51:06.820 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-17T14:51:06.820 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-17T14:51:06.820 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-17T14:51:06.820 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-17T14:51:06.820 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-17T14:51:06.820 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-17T14:51:06.820 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-17T14:51:06.820 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:06.820 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:06.887 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 401 kB/s | 548 kB 00:01 2024-05-17T14:51:07.055 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-cephadm 2024-05-17T14:51:07.087 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 83 kB/s | 51 kB 00:00 2024-05-17T14:51:07.091 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:07.091 INFO:teuthology.orchestra.run.smithi134.stdout:Total 3.4 MB/s | 7.0 MB 00:02 2024-05-17T14:51:07.164 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-05-17T14:51:07.175 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-05-17T14:51:07.175 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-05-17T14:51:07.273 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-05-17T14:51:07.273 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-05-17T14:51:07.588 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-05-17T14:51:07.658 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:41 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:51:07.758 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-17T14:51:07.788 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:51:07.789 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:07.789 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-17T14:51:07.789 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:07.789 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-17T14:51:07.789 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 143 k 2024-05-17T14:51:07.790 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-17T14:51:07.790 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-17T14:51:07.790 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-17T14:51:07.790 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-17T14:51:07.790 INFO:teuthology.orchestra.run.smithi195.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-17T14:51:07.790 INFO:teuthology.orchestra.run.smithi195.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-17T14:51:07.790 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:07.790 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-17T14:51:07.790 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:07.790 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2024-05-17T14:51:07.790 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:07.791 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 7.0 M 2024-05-17T14:51:07.791 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 33 M 2024-05-17T14:51:07.791 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-17T14:51:07.814 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-17T14:51:08.348 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-17T14:51:08.393 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-05-17T14:51:08.451 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-17T14:51:08.513 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-17T14:51:08.543 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 6/6 2024-05-17T14:51:08.576 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 6.0 MB 00:00 2024-05-17T14:51:08.578 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] python3-asyncssh-2.13.2-5.el9.noarch.rpm: Curl error (60): SSL peer certificate or SSH remote key was not OK for https://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-asyncssh-2.13.2-5.el9.noarch.rpm [SSL certificate problem: certificate has expired] 2024-05-17T14:51:08.635 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): ceph-mgr-cephadm-19.0.0-3735.gee59d86a.e 376 kB/s | 143 kB 00:00 2024-05-17T14:51:08.704 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] python3-natsort-7.1.1-5.el9.noarch.rpm: Curl error (60): SSL peer certificate or SSH remote key was not OK for https://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-natsort-7.1.1-5.el9.noarch.rpm [SSL certificate problem: certificate has expired] 2024-05-17T14:51:08.778 INFO:teuthology.orchestra.run.smithi195.stdout:[MIRROR] python3-typing-extensions-4.4.0-2.el9.noarch.rpm: Curl error (60): SSL peer certificate or SSH remote key was not OK for https://mirror-mci.yuki.net.uk/fedora-epel/9/Everything/x86_64/Packages/p/python3-typing-extensions-4.4.0-2.el9.noarch.rpm [SSL certificate problem: certificate has expired] 2024-05-17T14:51:08.918 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 170 kB/s | 58 kB 00:00 2024-05-17T14:51:08.977 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 160 kB/s | 51 kB 00:00 2024-05-17T14:51:09.027 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 864 kB/s | 548 kB 00:00 2024-05-17T14:51:09.028 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:09.028 INFO:teuthology.orchestra.run.smithi195.stdout:Total 5.6 MB/s | 7.0 MB 00:01 2024-05-17T14:51:09.099 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-17T14:51:09.110 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-17T14:51:09.110 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-17T14:51:09.209 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-17T14:51:09.210 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-17T14:51:09.343 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 6/6 2024-05-17T14:51:09.343 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 1/6 2024-05-17T14:51:09.343 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-17T14:51:09.343 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-17T14:51:09.343 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-17T14:51:09.343 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-17T14:51:09.525 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-17T14:51:09.607 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-17T14:51:09.607 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:09.608 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-05-17T14:51:09.608 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:51:09.608 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-17T14:51:09.608 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-17T14:51:09.608 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-17T14:51:09.608 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-17T14:51:09.608 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-17T14:51:09.608 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:09.608 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:51:09.714 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-17T14:51:09.780 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-17T14:51:09.872 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse 2024-05-17T14:51:10.316 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-17T14:51:10.400 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-17T14:51:10.447 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:41 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:51:10.462 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-17T14:51:10.491 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 6/6 2024-05-17T14:51:10.576 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:51:10.577 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:10.577 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-05-17T14:51:10.577 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:10.577 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-05-17T14:51:10.577 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 890 k 2024-05-17T14:51:10.577 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-05-17T14:51:10.577 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-17T14:51:10.577 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:10.577 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-05-17T14:51:10.577 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:10.577 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-05-17T14:51:10.578 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:10.578 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 970 k 2024-05-17T14:51:10.578 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.8 M 2024-05-17T14:51:10.578 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-05-17T14:51:10.823 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 658 kB/s | 80 kB 00:00 2024-05-17T14:51:11.006 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-fuse-19.0.0-3735.gee59d86a.el9.x86_ 2.8 MB/s | 890 kB 00:00 2024-05-17T14:51:11.007 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:11.007 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.2 MB/s | 970 kB 00:00 2024-05-17T14:51:11.020 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-05-17T14:51:11.043 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-05-17T14:51:11.043 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-05-17T14:51:11.126 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-05-17T14:51:11.127 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-05-17T14:51:11.235 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 6/6 2024-05-17T14:51:11.235 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 1/6 2024-05-17T14:51:11.235 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-17T14:51:11.235 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-17T14:51:11.235 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-17T14:51:11.235 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-17T14:51:11.313 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-05-17T14:51:11.444 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-17T14:51:11.493 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-17T14:51:11.494 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:11.494 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-17T14:51:11.494 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:51:11.494 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-17T14:51:11.494 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-17T14:51:11.494 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-17T14:51:11.494 INFO:teuthology.orchestra.run.smithi195.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-17T14:51:11.494 INFO:teuthology.orchestra.run.smithi195.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-17T14:51:11.494 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:11.494 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:11.523 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:19.0.0-3735.gee59d86a.el9.x86_64 2/2 2024-05-17T14:51:11.766 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-fuse 2024-05-17T14:51:12.241 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3735.gee59d86a.el9.x86_64 2/2 2024-05-17T14:51:12.241 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:19.0.0-3735.gee59d86a.el9.x86_64 1/2 2024-05-17T14:51:12.335 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:46 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:51:12.457 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:51:12.458 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:12.458 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-17T14:51:12.458 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:12.458 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-17T14:51:12.458 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 890 k 2024-05-17T14:51:12.458 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-17T14:51:12.458 INFO:teuthology.orchestra.run.smithi195.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-17T14:51:12.459 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:12.459 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-17T14:51:12.459 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:12.459 INFO:teuthology.orchestra.run.smithi195.stdout:Install 2 Packages 2024-05-17T14:51:12.459 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:12.459 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 970 k 2024-05-17T14:51:12.459 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 2.8 M 2024-05-17T14:51:12.460 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-17T14:51:12.525 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-17T14:51:12.525 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:12.525 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-05-17T14:51:12.525 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:19.0.0-3735.gee59d86a.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-17T14:51:12.525 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:12.525 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:51:12.803 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-volume 2024-05-17T14:51:13.021 INFO:teuthology.orchestra.run.smithi195.stdout:(1/2): ceph-fuse-19.0.0-3735.gee59d86a.el9.x86_ 3.0 MB/s | 890 kB 00:00 2024-05-17T14:51:13.390 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:44 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:51:13.421 INFO:teuthology.orchestra.run.smithi195.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 117 kB/s | 80 kB 00:00 2024-05-17T14:51:13.421 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:13.422 INFO:teuthology.orchestra.run.smithi195.stdout:Total 1.0 MB/s | 970 kB 00:00 2024-05-17T14:51:13.433 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-17T14:51:13.454 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-17T14:51:13.455 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-17T14:51:13.513 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:51:13.514 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-05-17T14:51:13.514 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:13.514 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-05-17T14:51:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 273 k 2024-05-17T14:51:13.514 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-05-17T14:51:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-17T14:51:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:13.514 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-05-17T14:51:13.514 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:13.515 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-05-17T14:51:13.515 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:13.515 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 350 k 2024-05-17T14:51:13.515 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.5 M 2024-05-17T14:51:13.515 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-05-17T14:51:13.537 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-17T14:51:13.537 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-17T14:51:13.730 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-17T14:51:13.819 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 422 kB/s | 77 kB 00:00 2024-05-17T14:51:13.864 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-17T14:51:13.885 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-volume-19.0.0-3735.gee59d86a.el9.no 1.1 MB/s | 273 kB 00:00 2024-05-17T14:51:13.886 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:13.886 INFO:teuthology.orchestra.run.smithi134.stdout:Total 942 kB/s | 350 kB 00:00 2024-05-17T14:51:13.898 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-05-17T14:51:13.907 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-05-17T14:51:13.907 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-05-17T14:51:13.939 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-fuse-2:19.0.0-3735.gee59d86a.el9.x86_64 2/2 2024-05-17T14:51:13.943 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-05-17T14:51:13.944 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-05-17T14:51:14.084 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-05-17T14:51:14.258 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-05-17T14:51:14.296 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-volume-2:19.0.0-3735.gee59d86a.el9.noarch 2/2 2024-05-17T14:51:14.312 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-volume-2:19.0.0-3735.gee59d86a.el9.noarch 2/2 2024-05-17T14:51:14.312 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-17T14:51:14.312 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-17T14:51:14.312 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:14.634 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3735.gee59d86a.el9.x86_64 2/2 2024-05-17T14:51:14.635 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-fuse-2:19.0.0-3735.gee59d86a.el9.x86_64 1/2 2024-05-17T14:51:14.886 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-17T14:51:14.886 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:14.886 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-17T14:51:14.886 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse-2:19.0.0-3735.gee59d86a.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-17T14:51:14.887 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:14.887 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:15.001 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-volume-2:19.0.0-3735.gee59d86a.el9.noarch 1/2 2024-05-17T14:51:15.183 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-volume 2024-05-17T14:51:15.300 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-05-17T14:51:15.300 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:15.300 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-05-17T14:51:15.300 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:51:15.300 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-17T14:51:15.300 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:15.301 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:51:15.608 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel 2024-05-17T14:51:15.766 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:49 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:51:15.898 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:51:15.898 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:15.898 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2024-05-17T14:51:15.899 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:15.899 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-17T14:51:15.899 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-volume noarch 2:19.0.0-3735.gee59d86a.el9 ceph-noarch 273 k 2024-05-17T14:51:15.899 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-05-17T14:51:15.899 INFO:teuthology.orchestra.run.smithi195.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-17T14:51:15.899 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:15.899 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-17T14:51:15.899 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:15.899 INFO:teuthology.orchestra.run.smithi195.stdout:Install 2 Packages 2024-05-17T14:51:15.899 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:15.900 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 350 k 2024-05-17T14:51:15.900 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 1.5 M 2024-05-17T14:51:15.900 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-17T14:51:16.197 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:47 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:51:16.323 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:51:16.324 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:16.324 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-05-17T14:51:16.324 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:16.324 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-05-17T14:51:16.324 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 127 k 2024-05-17T14:51:16.324 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:16.324 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-05-17T14:51:16.324 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:16.324 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-05-17T14:51:16.324 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:16.325 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 127 k 2024-05-17T14:51:16.325 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 456 k 2024-05-17T14:51:16.325 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-05-17T14:51:16.518 INFO:teuthology.orchestra.run.smithi195.stdout:(1/2): ceph-volume-19.0.0-3735.gee59d86a.el9.no 476 kB/s | 273 kB 00:00 2024-05-17T14:51:16.859 INFO:teuthology.orchestra.run.smithi195.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 84 kB/s | 77 kB 00:00 2024-05-17T14:51:16.860 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:16.861 INFO:teuthology.orchestra.run.smithi195.stdout:Total 364 kB/s | 350 kB 00:00 2024-05-17T14:51:16.874 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-17T14:51:16.884 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-17T14:51:16.884 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-17T14:51:16.918 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-17T14:51:16.918 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-17T14:51:17.064 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-17T14:51:17.232 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-19.0.0-3735.gee59d86a.el9.x86_64 140 kB/s | 127 kB 00:00 2024-05-17T14:51:17.232 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:17.233 INFO:teuthology.orchestra.run.smithi134.stdout:Total 139 kB/s | 127 kB 00:00 2024-05-17T14:51:17.233 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-05-17T14:51:17.239 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-05-17T14:51:17.239 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-05-17T14:51:17.259 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-05-17T14:51:17.287 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-volume-2:19.0.0-3735.gee59d86a.el9.noarch 2/2 2024-05-17T14:51:17.304 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-volume-2:19.0.0-3735.gee59d86a.el9.noarch 2/2 2024-05-17T14:51:17.304 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-17T14:51:17.304 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-17T14:51:17.304 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:17.304 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-05-17T14:51:17.304 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-05-17T14:51:17.463 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-05-17T14:51:17.571 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:18.030 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:18.053 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-volume-2:19.0.0-3735.gee59d86a.el9.noarch 1/2 2024-05-17T14:51:18.276 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:18.276 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:18.276 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-05-17T14:51:18.276 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:51:18.277 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:18.277 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:51:18.370 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-05-17T14:51:18.370 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:18.370 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-17T14:51:18.371 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-volume-2:19.0.0-3735.gee59d86a.el9.noarch 2024-05-17T14:51:18.371 INFO:teuthology.orchestra.run.smithi195.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-17T14:51:18.371 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:18.371 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:18.525 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2 2024-05-17T14:51:18.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados-devel 2024-05-17T14:51:19.115 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:50 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:51:19.199 INFO:teuthology.orchestra.run.smithi134.stdout:Package libcephfs2-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:51:19.242 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:51:19.242 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-05-17T14:51:19.242 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:51:19.264 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:53 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:51:19.320 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs-devel 2024-05-17T14:51:19.388 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:51:19.389 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:19.389 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-05-17T14:51:19.389 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:19.389 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-17T14:51:19.390 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 127 k 2024-05-17T14:51:19.390 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:19.390 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-17T14:51:19.390 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:19.390 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-05-17T14:51:19.390 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:19.390 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 127 k 2024-05-17T14:51:19.390 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 456 k 2024-05-17T14:51:19.390 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-17T14:51:19.724 INFO:teuthology.orchestra.run.smithi195.stdout:librados-devel-19.0.0-3735.gee59d86a.el9.x86_64 379 kB/s | 127 kB 00:00 2024-05-17T14:51:19.725 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:19.725 INFO:teuthology.orchestra.run.smithi195.stdout:Total 378 kB/s | 127 kB 00:00 2024-05-17T14:51:19.725 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-17T14:51:19.730 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-17T14:51:19.730 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-17T14:51:19.794 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-17T14:51:19.795 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-17T14:51:19.911 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:50 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:51:19.960 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-17T14:51:20.033 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:51:20.034 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:20.035 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-05-17T14:51:20.035 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:20.035 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-05-17T14:51:20.035 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 32 k 2024-05-17T14:51:20.035 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:20.035 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-05-17T14:51:20.035 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:20.035 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-05-17T14:51:20.035 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:20.035 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 32 k 2024-05-17T14:51:20.035 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 142 k 2024-05-17T14:51:20.035 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-05-17T14:51:20.055 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librados-devel-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:20.202 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs-devel-19.0.0-3735.gee59d86a.el9.x86_6 190 kB/s | 32 kB 00:00 2024-05-17T14:51:20.202 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:20.202 INFO:teuthology.orchestra.run.smithi134.stdout:Total 189 kB/s | 32 kB 00:00 2024-05-17T14:51:20.203 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-05-17T14:51:20.206 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-05-17T14:51:20.206 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-05-17T14:51:20.220 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-05-17T14:51:20.221 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-05-17T14:51:20.305 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-05-17T14:51:20.399 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:20.485 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados-devel-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:20.712 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados-devel-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:20.713 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:20.713 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-17T14:51:20.713 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:51:20.713 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:20.713 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:20.811 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:20.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs2 2024-05-17T14:51:21.023 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:21.023 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:21.023 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-05-17T14:51:21.023 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:51:21.023 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:21.023 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:51:21.207 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2 2024-05-17T14:51:21.559 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:55 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:51:21.642 INFO:teuthology.orchestra.run.smithi195.stdout:Package libcephfs2-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:51:21.685 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:51:21.686 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-17T14:51:21.686 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:21.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs-devel 2024-05-17T14:51:21.790 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:52 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:51:21.871 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:51:21.912 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:51:21.913 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-05-17T14:51:21.913 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:51:21.987 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1 2024-05-17T14:51:22.347 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:56 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:51:22.470 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:51:22.471 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:22.471 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-05-17T14:51:22.471 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:22.471 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-17T14:51:22.471 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 32 k 2024-05-17T14:51:22.471 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:22.471 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-17T14:51:22.471 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:22.471 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-05-17T14:51:22.471 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:22.472 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 32 k 2024-05-17T14:51:22.472 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 142 k 2024-05-17T14:51:22.472 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-17T14:51:22.555 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:53 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:51:22.636 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:51:22.644 INFO:teuthology.orchestra.run.smithi195.stdout:libcephfs-devel-19.0.0-3735.gee59d86a.el9.x86_6 184 kB/s | 32 kB 00:00 2024-05-17T14:51:22.644 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:22.645 INFO:teuthology.orchestra.run.smithi195.stdout:Total 183 kB/s | 32 kB 00:00 2024-05-17T14:51:22.645 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-17T14:51:22.648 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-17T14:51:22.648 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-17T14:51:22.660 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-17T14:51:22.660 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-17T14:51:22.678 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:51:22.679 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-05-17T14:51:22.679 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:51:22.742 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-17T14:51:22.757 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados 2024-05-17T14:51:22.849 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs-devel-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:23.241 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:23.334 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:54 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:51:23.416 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rados-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:51:23.457 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:51:23.458 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs-devel-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:23.458 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:23.458 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-17T14:51:23.458 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:51:23.458 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:23.458 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:23.459 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-05-17T14:51:23.459 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:51:23.536 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw 2024-05-17T14:51:23.635 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados2 2024-05-17T14:51:24.107 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:55 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:51:24.187 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rgw-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:51:24.210 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:58 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:51:24.229 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:51:24.230 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-05-17T14:51:24.230 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:51:24.292 INFO:teuthology.orchestra.run.smithi195.stdout:Package librados2-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:51:24.310 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs 2024-05-17T14:51:24.334 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:51:24.335 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-17T14:51:24.335 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:24.411 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librbd1 2024-05-17T14:51:24.883 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:55 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:51:24.963 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-cephfs-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:51:24.985 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:58 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:51:25.004 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:51:25.005 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-05-17T14:51:25.005 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:51:25.065 INFO:teuthology.orchestra.run.smithi195.stdout:Package librbd1-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:51:25.079 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd 2024-05-17T14:51:25.108 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:51:25.109 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-17T14:51:25.109 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:25.183 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rados 2024-05-17T14:51:25.645 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:56 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:51:25.725 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rbd-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:51:25.757 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:59 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:51:25.767 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:51:25.768 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-05-17T14:51:25.768 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:51:25.839 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rados-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:51:25.843 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse 2024-05-17T14:51:25.880 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:51:25.882 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-17T14:51:25.882 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:25.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rgw 2024-05-17T14:51:26.412 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:57 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:51:26.520 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:00 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:51:26.534 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:51:26.534 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:26.535 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-05-17T14:51:26.535 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:26.535 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-05-17T14:51:26.535 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 85 k 2024-05-17T14:51:26.535 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:26.535 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-05-17T14:51:26.535 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:26.535 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-05-17T14:51:26.535 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:26.536 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 85 k 2024-05-17T14:51:26.536 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 231 k 2024-05-17T14:51:26.536 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-05-17T14:51:26.603 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rgw-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:51:26.644 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:51:26.645 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-17T14:51:26.646 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:26.716 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-cephfs 2024-05-17T14:51:26.738 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-19.0.0-3735.gee59d86a.el9.x86_64.rpm 422 kB/s | 85 kB 00:00 2024-05-17T14:51:26.738 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:26.739 INFO:teuthology.orchestra.run.smithi134.stdout:Total 420 kB/s | 85 kB 00:00 2024-05-17T14:51:26.739 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-05-17T14:51:26.746 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-05-17T14:51:26.746 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-05-17T14:51:26.809 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-05-17T14:51:26.809 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-05-17T14:51:26.959 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-05-17T14:51:27.060 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:27.289 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:01 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:51:27.369 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-cephfs-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:51:27.410 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:51:27.411 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-17T14:51:27.411 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:27.433 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:27.478 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rbd 2024-05-17T14:51:27.688 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:27.689 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:27.689 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-05-17T14:51:27.689 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:51:27.689 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:27.689 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:51:27.920 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror 2024-05-17T14:51:28.051 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:02 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:51:28.133 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rbd-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:51:28.175 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:51:28.176 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-17T14:51:28.176 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:28.244 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-fuse 2024-05-17T14:51:28.496 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:59 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:51:28.620 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:51:28.620 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:28.620 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-05-17T14:51:28.620 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:28.620 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-05-17T14:51:28.621 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 3.1 M 2024-05-17T14:51:28.621 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:28.621 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-05-17T14:51:28.621 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:28.621 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-05-17T14:51:28.621 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:28.621 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.1 M 2024-05-17T14:51:28.622 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 13 M 2024-05-17T14:51:28.622 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-05-17T14:51:28.824 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:02 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:51:28.948 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:51:28.948 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:28.949 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-05-17T14:51:28.949 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:28.949 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-17T14:51:28.949 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 85 k 2024-05-17T14:51:28.949 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:28.949 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-17T14:51:28.949 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:28.949 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-05-17T14:51:28.949 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:28.950 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 85 k 2024-05-17T14:51:28.950 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 231 k 2024-05-17T14:51:28.950 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-17T14:51:29.164 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-fuse-19.0.0-3735.gee59d86a.el9.x86_64.rpm 399 kB/s | 85 kB 00:00 2024-05-17T14:51:29.164 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:29.165 INFO:teuthology.orchestra.run.smithi195.stdout:Total 397 kB/s | 85 kB 00:00 2024-05-17T14:51:29.165 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-17T14:51:29.172 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-17T14:51:29.172 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-17T14:51:29.235 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-17T14:51:29.235 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-17T14:51:29.379 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-17T14:51:29.487 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-fuse-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:29.527 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-19.0.0-3735.gee59d86a.el9.x86_64.rpm 3.4 MB/s | 3.1 MB 00:00 2024-05-17T14:51:29.528 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:29.528 INFO:teuthology.orchestra.run.smithi134.stdout:Total 3.4 MB/s | 3.1 MB 00:00 2024-05-17T14:51:29.528 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-05-17T14:51:29.537 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-05-17T14:51:29.537 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-05-17T14:51:29.616 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-05-17T14:51:29.617 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-05-17T14:51:29.884 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:30.047 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-05-17T14:51:30.112 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:30.137 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-fuse-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:30.137 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:30.137 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-17T14:51:30.137 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:51:30.137 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:30.137 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:30.138 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:30.138 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-17T14:51:30.138 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-17T14:51:30.138 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-17T14:51:30.138 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-17T14:51:30.138 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:30.391 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-mirror 2024-05-17T14:51:30.977 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:04 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:51:31.107 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:51:31.107 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:31.107 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2024-05-17T14:51:31.107 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:31.107 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-17T14:51:31.107 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 3.1 M 2024-05-17T14:51:31.107 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:31.108 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-17T14:51:31.108 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:31.108 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-05-17T14:51:31.108 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:31.108 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 3.1 M 2024-05-17T14:51:31.108 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 13 M 2024-05-17T14:51:31.108 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-17T14:51:31.171 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:31.171 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:31.171 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-05-17T14:51:31.171 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:51:31.171 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:31.171 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:51:31.435 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd 2024-05-17T14:51:31.996 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-mirror-19.0.0-3735.gee59d86a.el9.x86_64.rpm 3.5 MB/s | 3.1 MB 00:00 2024-05-17T14:51:31.997 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:31.997 INFO:teuthology.orchestra.run.smithi195.stdout:Total 3.5 MB/s | 3.1 MB 00:00 2024-05-17T14:51:31.997 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-17T14:51:32.006 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-17T14:51:32.007 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-17T14:51:32.018 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:03 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:51:32.087 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-17T14:51:32.087 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-17T14:51:32.141 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:51:32.141 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:32.142 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-05-17T14:51:32.142 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:32.142 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-05-17T14:51:32.142 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 170 k 2024-05-17T14:51:32.142 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:32.142 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-05-17T14:51:32.142 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-05-17T14:51:32.142 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-05-17T14:51:32.142 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:32.142 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 170 k 2024-05-17T14:51:32.142 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 494 k 2024-05-17T14:51:32.142 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-05-17T14:51:32.374 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-19.0.0-3735.gee59d86a.el9.x86_64.rpm 738 kB/s | 170 kB 00:00 2024-05-17T14:51:32.375 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:32.375 INFO:teuthology.orchestra.run.smithi134.stdout:Total 733 kB/s | 170 kB 00:00 2024-05-17T14:51:32.375 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-05-17T14:51:32.382 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-05-17T14:51:32.382 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-05-17T14:51:32.447 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-05-17T14:51:32.447 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-05-17T14:51:32.519 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-17T14:51:32.581 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-mirror-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:32.599 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-05-17T14:51:32.605 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:32.606 INFO:teuthology.orchestra.run.smithi195.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-17T14:51:32.606 INFO:teuthology.orchestra.run.smithi195.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-17T14:51:32.606 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-17T14:51:32.606 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-17T14:51:32.606 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:32.704 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:33.114 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:33.379 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:33.379 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:33.379 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-05-17T14:51:33.379 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:51:33.379 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:33.380 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:51:33.625 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror 2024-05-17T14:51:33.665 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-mirror-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:33.666 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:33.666 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-17T14:51:33.666 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:51:33.666 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:33.666 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:33.903 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-nbd 2024-05-17T14:51:34.209 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:05 ago on Fri 17 May 2024 02:49:29 PM UTC. 2024-05-17T14:51:34.291 INFO:teuthology.orchestra.run.smithi134.stdout:Package rbd-mirror-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:51:34.334 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-05-17T14:51:34.335 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-05-17T14:51:34.335 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-05-17T14:51:34.407 DEBUG:teuthology.parallel:result is None 2024-05-17T14:51:34.493 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:08 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:51:34.617 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:51:34.617 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:34.618 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-05-17T14:51:34.618 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:34.618 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-05-17T14:51:34.618 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd x86_64 2:19.0.0-3735.gee59d86a.el9 ceph 170 k 2024-05-17T14:51:34.618 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:34.618 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-05-17T14:51:34.618 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-05-17T14:51:34.618 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-05-17T14:51:34.618 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:34.618 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 170 k 2024-05-17T14:51:34.618 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 494 k 2024-05-17T14:51:34.618 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-05-17T14:51:34.843 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-nbd-19.0.0-3735.gee59d86a.el9.x86_64.rpm 760 kB/s | 170 kB 00:00 2024-05-17T14:51:34.844 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-05-17T14:51:34.844 INFO:teuthology.orchestra.run.smithi195.stdout:Total 755 kB/s | 170 kB 00:00 2024-05-17T14:51:34.844 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-05-17T14:51:34.851 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-05-17T14:51:34.851 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-05-17T14:51:34.914 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-05-17T14:51:34.914 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-05-17T14:51:35.078 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-05-17T14:51:35.177 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-nbd-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:35.599 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:35.864 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-nbd-2:19.0.0-3735.gee59d86a.el9.x86_64 1/1 2024-05-17T14:51:35.865 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:35.865 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-05-17T14:51:35.865 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd-2:19.0.0-3735.gee59d86a.el9.x86_64 2024-05-17T14:51:35.865 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:35.865 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:36.116 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-mirror 2024-05-17T14:51:36.709 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:10 ago on Fri 17 May 2024 02:49:26 PM UTC. 2024-05-17T14:51:36.796 INFO:teuthology.orchestra.run.smithi195.stdout:Package rbd-mirror-2:19.0.0-3735.gee59d86a.el9.x86_64 is already installed. 2024-05-17T14:51:36.842 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-05-17T14:51:36.843 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2024-05-17T14:51:36.843 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-05-17T14:51:36.921 DEBUG:teuthology.parallel:result is None 2024-05-17T14:51:36.922 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ee59d86aad04089481e55c1a15aa5871339935b4 2024-05-17T14:51:37.071 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-17T14:51:37.097 INFO:teuthology.orchestra.run.smithi134.stdout:19.0.0-3735.gee59d86a.el9 2024-05-17T14:51:37.097 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3735.gee59d86a.el9 2024-05-17T14:51:37.097 INFO:teuthology.task.install:The correct ceph version 19.0.0-3735.gee59d86a is installed. 2024-05-17T14:51:37.099 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ee59d86aad04089481e55c1a15aa5871339935b4 2024-05-17T14:51:37.235 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-17T14:51:37.257 INFO:teuthology.orchestra.run.smithi195.stdout:19.0.0-3735.gee59d86a.el9 2024-05-17T14:51:37.257 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3735.gee59d86a.el9 2024-05-17T14:51:37.258 INFO:teuthology.task.install:The correct ceph version 19.0.0-3735.gee59d86a is installed. 2024-05-17T14:51:37.260 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-17T14:51:37.260 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:37.260 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-17T14:51:37.290 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:51:37.290 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-17T14:51:37.322 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-17T14:51:37.322 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:37.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-05-17T14:51:37.390 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-17T14:51:37.458 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:51:37.458 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/daemon-helper 2024-05-17T14:51:37.483 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-17T14:51:37.548 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-17T14:51:37.548 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:37.548 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-17T14:51:37.615 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-17T14:51:37.683 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:51:37.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-17T14:51:37.708 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-17T14:51:37.773 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-17T14:51:37.773 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:37.773 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-05-17T14:51:37.802 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-17T14:51:37.870 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:51:37.871 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/stdin-killer 2024-05-17T14:51:37.895 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-17T14:51:37.960 INFO:teuthology.run_tasks:Running task ceph... 2024-05-17T14:51:38.102 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-17T14:51:38.102 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /var/log/ceph 2024-05-17T14:51:38.104 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /var/log/ceph 2024-05-17T14:51:38.131 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-17T14:51:38.131 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-17T14:51:38.174 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-17T14:51:38.201 INFO:tasks.ceph:Creating extra log directories... 2024-05-17T14:51:38.201 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-17T14:51:38.243 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-17T14:51:38.281 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-05-17T14:51:38.281 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-17T14:51:38.281 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7711219', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing', 'sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 3754, 'sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4', 'sleep_before_teardown': 0, 'subset': '56/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4', 'targets': {'smithi134.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHYe1ZGnxZjX7+sQweRvnThF1q47PodBSwam7dcyTHgN8O9FeALEpBU/YCmwo/S2di3fktueeGSo22ojmFO2hnY=', 'smithi195.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHOwvwP+xWhFfbARMc1jjL85g4dIJywrcGaxZwuxpRdYo+R7TPpbE+bquiE8iV2Wra/zwrM6BG46opM+s86KYvw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'wip-fix-extra-system-packages', 'teuthology_sha1': 'cad7edd8d899a7c59a48f7da3294fca4cb778f64', 'timestamp': '2024-05-17_14:29:27', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-17T14:51:38.282 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-05-17T14:51:38.312 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-05-17T14:51:38.327 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/run/ceph 2024-05-17T14:51:38.369 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/run/ceph 2024-05-17T14:51:38.393 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:38.393 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2024-05-17T14:51:38.449 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-17T14:51:38.449 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_1 2024-05-17T14:51:38.505 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-17T14:51:38.505 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-17T14:51:38.506 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-17T14:51:38.506 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-17T14:51:38.506 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-05-17T14:51:38.506 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-05-17 14:51:30.413534184 +0000 2024-05-17T14:51:38.506 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-05-17 14:48:20.569361379 +0000 2024-05-17T14:51:38.506 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-05-17 14:48:20.569361379 +0000 2024-05-17T14:51:38.506 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-05-17 14:48:20.569361379 +0000 2024-05-17T14:51:38.506 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-17T14:51:38.570 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-05-17T14:51:38.570 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-05-17T14:51:38.570 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000181797 s, 2.8 MB/s 2024-05-17T14:51:38.571 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-17T14:51:38.629 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_2 2024-05-17T14:51:38.684 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-17T14:51:38.685 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-17T14:51:38.685 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-05-17T14:51:38.685 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-17T14:51:38.685 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-05-17T14:51:38.685 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-05-17 14:51:30.413534184 +0000 2024-05-17T14:51:38.685 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-05-17 14:48:20.930352700 +0000 2024-05-17T14:51:38.685 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-05-17 14:48:20.930352700 +0000 2024-05-17T14:51:38.685 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-05-17 14:48:20.930352700 +0000 2024-05-17T14:51:38.685 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-17T14:51:38.748 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-05-17T14:51:38.748 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-05-17T14:51:38.748 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.00020839 s, 2.5 MB/s 2024-05-17T14:51:38.750 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-17T14:51:38.806 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_3 2024-05-17T14:51:38.860 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-17T14:51:38.861 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-17T14:51:38.861 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-17T14:51:38.861 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-17T14:51:38.861 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-05-17T14:51:38.861 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-05-17 14:51:30.414534160 +0000 2024-05-17T14:51:38.861 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-05-17 14:48:21.254344910 +0000 2024-05-17T14:51:38.861 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-05-17 14:48:21.254344910 +0000 2024-05-17T14:51:38.861 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-05-17 14:48:21.254344910 +0000 2024-05-17T14:51:38.861 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-17T14:51:38.924 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-05-17T14:51:38.924 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-05-17T14:51:38.924 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000217113 s, 2.4 MB/s 2024-05-17T14:51:38.925 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-17T14:51:38.982 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_4 2024-05-17T14:51:39.038 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-17T14:51:39.039 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-17T14:51:39.039 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-05-17T14:51:39.039 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-17T14:51:39.039 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-05-17T14:51:39.039 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-05-17 14:51:30.414534160 +0000 2024-05-17T14:51:39.039 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-05-17 14:48:21.623336039 +0000 2024-05-17T14:51:39.039 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-05-17 14:48:21.623336039 +0000 2024-05-17T14:51:39.039 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-05-17 14:48:21.623336039 +0000 2024-05-17T14:51:39.039 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-17T14:51:39.101 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-05-17T14:51:39.102 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-05-17T14:51:39.102 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000224163 s, 2.3 MB/s 2024-05-17T14:51:39.103 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-17T14:51:39.159 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-17T14:51:39.159 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-05-17T14:51:39.159 INFO:tasks.ceph:Generating config... 2024-05-17T14:51:39.161 INFO:tasks.ceph:[client] debug rbd = 20 2024-05-17T14:51:39.161 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-05-17T14:51:39.161 INFO:tasks.ceph:[client] log to stderr = False 2024-05-17T14:51:39.161 INFO:tasks.ceph:[client] rbd default features = 125 2024-05-17T14:51:39.161 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-05-17T14:51:39.161 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:51:39.161 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:51:39.161 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:51:39.161 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:51:39.161 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:51:39.162 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:51:39.162 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:51:39.162 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:51:39.162 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-05-17T14:51:39.162 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-05-17T14:51:39.162 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-05-17T14:51:39.162 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-17T14:51:39.162 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-17T14:51:39.162 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-17T14:51:39.162 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-17T14:51:39.162 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-17T14:51:39.163 INFO:tasks.ceph:[osd] bdev async discard = True 2024-05-17T14:51:39.163 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-05-17T14:51:39.163 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-17T14:51:39.163 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-17T14:51:39.163 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-05-17T14:51:39.163 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-05-17T14:51:39.163 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-17T14:51:39.163 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-17T14:51:39.163 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-05-17T14:51:39.163 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-17T14:51:39.163 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-17T14:51:39.163 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-17T14:51:39.164 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-17T14:51:39.164 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-05-17T14:51:39.164 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-17T14:51:39.164 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-05-17T14:51:39.164 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-05-17T14:51:39.252 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.keyring 2024-05-17T14:51:39.255 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-05-17T14:51:39.297 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-05-17T14:51:39.362 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.134')] 2024-05-17T14:51:39.363 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.134', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-05-17T14:51:39.364 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:39.365 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-17T14:51:39.421 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.134 --print /home/ubuntu/cephtest/cluster1.monmap 2024-05-17T14:51:39.499 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-05-17T14:51:39.500 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: generated fsid 5b29e51e-670a-453e-8d22-58d6f74d2947 2024-05-17T14:51:39.500 INFO:teuthology.orchestra.run.smithi134.stdout:setting min_mon_release = quincy 2024-05-17T14:51:39.500 INFO:teuthology.orchestra.run.smithi134.stdout:epoch 0 2024-05-17T14:51:39.500 INFO:teuthology.orchestra.run.smithi134.stdout:fsid 5b29e51e-670a-453e-8d22-58d6f74d2947 2024-05-17T14:51:39.500 INFO:teuthology.orchestra.run.smithi134.stdout:last_changed 2024-05-17T14:51:39.500287+0000 2024-05-17T14:51:39.500 INFO:teuthology.orchestra.run.smithi134.stdout:created 2024-05-17T14:51:39.500287+0000 2024-05-17T14:51:39.500 INFO:teuthology.orchestra.run.smithi134.stdout:min_mon_release 17 (quincy) 2024-05-17T14:51:39.500 INFO:teuthology.orchestra.run.smithi134.stdout:election_strategy: 1 2024-05-17T14:51:39.500 INFO:teuthology.orchestra.run.smithi134.stdout:0: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.a 2024-05-17T14:51:39.500 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-05-17T14:51:39.501 DEBUG:teuthology.orchestra.run.smithi134:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-17T14:51:39.556 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 5b29e51e-670a-453e-8d22-58d6f74d2947... 2024-05-17T14:51:39.557 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-05-17T14:51:39.599 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-05-17T14:51:39.638 INFO:teuthology.orchestra.run.smithi134.stdout:[global] 2024-05-17T14:51:39.639 INFO:teuthology.orchestra.run.smithi134.stdout: chdir = "" 2024-05-17T14:51:39.639 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-17T14:51:39.639 INFO:teuthology.orchestra.run.smithi134.stdout: auth supported = cephx 2024-05-17T14:51:39.639 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.639 INFO:teuthology.orchestra.run.smithi134.stdout: filestore xattr use omap = true 2024-05-17T14:51:39.639 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.639 INFO:teuthology.orchestra.run.smithi134.stdout: mon clock drift allowed = 1.000 2024-05-17T14:51:39.639 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.639 INFO:teuthology.orchestra.run.smithi134.stdout: osd crush chooseleaf type = 0 2024-05-17T14:51:39.639 INFO:teuthology.orchestra.run.smithi134.stdout: auth debug = true 2024-05-17T14:51:39.639 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.640 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on old message = true 2024-05-17T14:51:39.640 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on bug = true 2024-05-17T14:51:39.640 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.640 INFO:teuthology.orchestra.run.smithi134.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-17T14:51:39.640 INFO:teuthology.orchestra.run.smithi134.stdout: mon pg warn max object skew = 0 2024-05-17T14:51:39.640 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.640 INFO:teuthology.orchestra.run.smithi134.stdout: # disable pg_autoscaler by default for new pools 2024-05-17T14:51:39.640 INFO:teuthology.orchestra.run.smithi134.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-17T14:51:39.640 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.640 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default size = 2 2024-05-17T14:51:39.640 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.640 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow primary affinity = true 2024-05-17T14:51:39.640 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow pg remap = true 2024-05-17T14:51:39.641 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on legacy crush tunables = false 2024-05-17T14:51:39.641 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on crush straw calc version zero = false 2024-05-17T14:51:39.641 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on no sortbitwise = false 2024-05-17T14:51:39.641 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on osd down out interval zero = false 2024-05-17T14:51:39.641 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on too few osds = false 2024-05-17T14:51:39.641 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-17T14:51:39.641 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-17T14:51:39.641 INFO:teuthology.orchestra.run.smithi134.stdout: mon_allow_pool_size_one = true 2024-05-17T14:51:39.642 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.642 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-17T14:51:39.642 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.642 INFO:teuthology.orchestra.run.smithi134.stdout: osd default data pool replay window = 5 2024-05-17T14:51:39.642 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.642 INFO:teuthology.orchestra.run.smithi134.stdout: mon allow pool delete = true 2024-05-17T14:51:39.642 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.642 INFO:teuthology.orchestra.run.smithi134.stdout: mon cluster log file level = debug 2024-05-17T14:51:39.642 INFO:teuthology.orchestra.run.smithi134.stdout: debug asserts on shutdown = true 2024-05-17T14:51:39.642 INFO:teuthology.orchestra.run.smithi134.stdout: mon health detail to clog = false 2024-05-17T14:51:39.642 INFO:teuthology.orchestra.run.smithi134.stdout: mon host = 172.21.15.134 2024-05-17T14:51:39.642 INFO:teuthology.orchestra.run.smithi134.stdout: mon client directed command retry = 5 2024-05-17T14:51:39.643 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on pool no app = False 2024-05-17T14:51:39.643 INFO:teuthology.orchestra.run.smithi134.stdout: ms inject socket failures = 5000 2024-05-17T14:51:39.643 INFO:teuthology.orchestra.run.smithi134.stdout: fsid = 5b29e51e-670a-453e-8d22-58d6f74d2947 2024-05-17T14:51:39.643 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.643 INFO:teuthology.orchestra.run.smithi134.stdout:[osd] 2024-05-17T14:51:39.643 INFO:teuthology.orchestra.run.smithi134.stdout: osd journal size = 100 2024-05-17T14:51:39.643 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.643 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub load threshold = 5.0 2024-05-17T14:51:39.643 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub max interval = 600 2024-05-17T14:51:39.643 INFO:teuthology.orchestra.run.smithi134.stdout: osd mclock profile = high_recovery_ops 2024-05-17T14:51:39.643 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.643 INFO:teuthology.orchestra.run.smithi134.stdout: osd recover clone overlap = true 2024-05-17T14:51:39.643 INFO:teuthology.orchestra.run.smithi134.stdout: osd recovery max chunk = 1048576 2024-05-17T14:51:39.644 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.644 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug shutdown = true 2024-05-17T14:51:39.644 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug op order = true 2024-05-17T14:51:39.644 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify stray on activate = true 2024-05-17T14:51:39.644 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug trim objects = true 2024-05-17T14:51:39.644 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.644 INFO:teuthology.orchestra.run.smithi134.stdout: osd open classes on start = true 2024-05-17T14:51:39.644 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug pg log writeout = true 2024-05-17T14:51:39.644 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.644 INFO:teuthology.orchestra.run.smithi134.stdout: osd deep scrub update digest min age = 30 2024-05-17T14:51:39.644 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.644 INFO:teuthology.orchestra.run.smithi134.stdout: osd map max advance = 10 2024-05-17T14:51:39.644 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.644 INFO:teuthology.orchestra.run.smithi134.stdout: journal zero on create = true 2024-05-17T14:51:39.645 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.645 INFO:teuthology.orchestra.run.smithi134.stdout: filestore ondisk finisher threads = 3 2024-05-17T14:51:39.645 INFO:teuthology.orchestra.run.smithi134.stdout: filestore apply finisher threads = 3 2024-05-17T14:51:39.645 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.645 INFO:teuthology.orchestra.run.smithi134.stdout: bdev debug aio = true 2024-05-17T14:51:39.645 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug misdirected ops = true 2024-05-17T14:51:39.645 INFO:teuthology.orchestra.run.smithi134.stdout: bdev async discard = True 2024-05-17T14:51:39.645 INFO:teuthology.orchestra.run.smithi134.stdout: bdev enable discard = True 2024-05-17T14:51:39.645 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore block size = 96636764160 2024-05-17T14:51:39.645 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore fsck on mount = True 2024-05-17T14:51:39.645 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluefs = 1/20 2024-05-17T14:51:39.645 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluestore = 1/20 2024-05-17T14:51:39.645 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-05-17T14:51:39.645 INFO:teuthology.orchestra.run.smithi134.stdout: debug osd = 20 2024-05-17T14:51:39.646 INFO:teuthology.orchestra.run.smithi134.stdout: debug rocksdb = 4/10 2024-05-17T14:51:39.646 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd backfillfull_ratio = 0.85 2024-05-17T14:51:39.646 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd full ratio = 0.9 2024-05-17T14:51:39.646 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd nearfull ratio = 0.8 2024-05-17T14:51:39.646 INFO:teuthology.orchestra.run.smithi134.stdout: osd failsafe full ratio = 0.95 2024-05-17T14:51:39.646 INFO:teuthology.orchestra.run.smithi134.stdout: osd memory target = 1610612736 2024-05-17T14:51:39.646 INFO:teuthology.orchestra.run.smithi134.stdout: osd objectstore = bluestore 2024-05-17T14:51:39.646 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.646 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr] 2024-05-17T14:51:39.646 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-05-17T14:51:39.647 INFO:teuthology.orchestra.run.smithi134.stdout: debug mgr = 20 2024-05-17T14:51:39.647 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-05-17T14:51:39.647 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-05-17T14:51:39.647 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-05-17T14:51:39.647 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-05-17T14:51:39.647 INFO:teuthology.orchestra.run.smithi134.stdout: mgr/telemetry/nag = false 2024-05-17T14:51:39.647 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.647 INFO:teuthology.orchestra.run.smithi134.stdout:[mon] 2024-05-17T14:51:39.647 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-05-17T14:51:39.647 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-05-17T14:51:39.647 INFO:teuthology.orchestra.run.smithi134.stdout: debug paxos = 20 2024-05-17T14:51:39.647 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-05-17T14:51:39.647 INFO:teuthology.orchestra.run.smithi134.stdout: mon data avail warn = 5 2024-05-17T14:51:39.647 INFO:teuthology.orchestra.run.smithi134.stdout: mon mgr mkfs grace = 240 2024-05-17T14:51:39.647 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-05-17T14:51:39.648 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd reporter subtree level = osd 2024-05-17T14:51:39.648 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd prime pg temp = true 2024-05-17T14:51:39.648 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-05-17T14:51:39.648 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.648 INFO:teuthology.orchestra.run.smithi134.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-17T14:51:39.648 INFO:teuthology.orchestra.run.smithi134.stdout: auth mon ticket ttl = 660 # 11m 2024-05-17T14:51:39.648 INFO:teuthology.orchestra.run.smithi134.stdout: auth service ticket ttl = 240 # 4m 2024-05-17T14:51:39.648 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.649 INFO:teuthology.orchestra.run.smithi134.stdout: # don't complain about insecure global_id in the test suite 2024-05-17T14:51:39.649 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-17T14:51:39.649 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-17T14:51:39.649 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.649 INFO:teuthology.orchestra.run.smithi134.stdout: # 1m isn't quite enough 2024-05-17T14:51:39.649 INFO:teuthology.orchestra.run.smithi134.stdout: mon_down_mkfs_grace = 2m 2024-05-17T14:51:39.649 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.649 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_filestore_osds = false 2024-05-17T14:51:39.649 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:39.649 INFO:teuthology.orchestra.run.smithi134.stdout:[client] 2024-05-17T14:51:39.649 INFO:teuthology.orchestra.run.smithi134.stdout: rgw cache enabled = true 2024-05-17T14:51:39.649 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable ops log = true 2024-05-17T14:51:39.649 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable usage log = true 2024-05-17T14:51:39.650 INFO:teuthology.orchestra.run.smithi134.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-17T14:51:39.650 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-17T14:51:39.650 INFO:teuthology.orchestra.run.smithi134.stdout: debug rbd = 20 2024-05-17T14:51:39.650 INFO:teuthology.orchestra.run.smithi134.stdout: debug rbd_mirror = 15 2024-05-17T14:51:39.650 INFO:teuthology.orchestra.run.smithi134.stdout: log to stderr = False 2024-05-17T14:51:39.650 INFO:teuthology.orchestra.run.smithi134.stdout: rbd default features = 125 2024-05-17T14:51:39.650 INFO:teuthology.orchestra.run.smithi134.stdout: rbd mirror image policy type = simple 2024-05-17T14:51:39.650 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.a] 2024-05-17T14:51:39.650 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.0] 2024-05-17T14:51:39.650 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:51:39.650 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:51:39.650 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.1] 2024-05-17T14:51:39.650 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:51:39.651 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:51:39.651 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.2] 2024-05-17T14:51:39.651 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:51:39.651 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:51:39.651 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.3] 2024-05-17T14:51:39.651 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:51:39.651 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:51:39.652 INFO:teuthology.orchestra.run.smithi195.stdout:[global] 2024-05-17T14:51:39.652 INFO:teuthology.orchestra.run.smithi195.stdout: chdir = "" 2024-05-17T14:51:39.652 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-17T14:51:39.652 INFO:teuthology.orchestra.run.smithi195.stdout: auth supported = cephx 2024-05-17T14:51:39.652 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.652 INFO:teuthology.orchestra.run.smithi195.stdout: filestore xattr use omap = true 2024-05-17T14:51:39.652 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.652 INFO:teuthology.orchestra.run.smithi195.stdout: mon clock drift allowed = 1.000 2024-05-17T14:51:39.652 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.652 INFO:teuthology.orchestra.run.smithi195.stdout: osd crush chooseleaf type = 0 2024-05-17T14:51:39.652 INFO:teuthology.orchestra.run.smithi195.stdout: auth debug = true 2024-05-17T14:51:39.652 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.652 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on old message = true 2024-05-17T14:51:39.652 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on bug = true 2024-05-17T14:51:39.653 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.653 INFO:teuthology.orchestra.run.smithi195.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-17T14:51:39.653 INFO:teuthology.orchestra.run.smithi195.stdout: mon pg warn max object skew = 0 2024-05-17T14:51:39.653 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.653 INFO:teuthology.orchestra.run.smithi195.stdout: # disable pg_autoscaler by default for new pools 2024-05-17T14:51:39.653 INFO:teuthology.orchestra.run.smithi195.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-17T14:51:39.653 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.653 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default size = 2 2024-05-17T14:51:39.653 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.653 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow primary affinity = true 2024-05-17T14:51:39.653 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow pg remap = true 2024-05-17T14:51:39.653 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on legacy crush tunables = false 2024-05-17T14:51:39.653 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on crush straw calc version zero = false 2024-05-17T14:51:39.653 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on no sortbitwise = false 2024-05-17T14:51:39.654 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on osd down out interval zero = false 2024-05-17T14:51:39.654 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on too few osds = false 2024-05-17T14:51:39.654 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-17T14:51:39.654 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-17T14:51:39.654 INFO:teuthology.orchestra.run.smithi195.stdout: mon_allow_pool_size_one = true 2024-05-17T14:51:39.654 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.654 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-17T14:51:39.654 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.654 INFO:teuthology.orchestra.run.smithi195.stdout: osd default data pool replay window = 5 2024-05-17T14:51:39.654 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.654 INFO:teuthology.orchestra.run.smithi195.stdout: mon allow pool delete = true 2024-05-17T14:51:39.654 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.654 INFO:teuthology.orchestra.run.smithi195.stdout: mon cluster log file level = debug 2024-05-17T14:51:39.655 INFO:teuthology.orchestra.run.smithi195.stdout: debug asserts on shutdown = true 2024-05-17T14:51:39.655 INFO:teuthology.orchestra.run.smithi195.stdout: mon health detail to clog = false 2024-05-17T14:51:39.655 INFO:teuthology.orchestra.run.smithi195.stdout: mon host = 172.21.15.134 2024-05-17T14:51:39.655 INFO:teuthology.orchestra.run.smithi195.stdout: mon client directed command retry = 5 2024-05-17T14:51:39.655 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on pool no app = False 2024-05-17T14:51:39.655 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject socket failures = 5000 2024-05-17T14:51:39.655 INFO:teuthology.orchestra.run.smithi195.stdout: fsid = 5b29e51e-670a-453e-8d22-58d6f74d2947 2024-05-17T14:51:39.655 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.655 INFO:teuthology.orchestra.run.smithi195.stdout:[osd] 2024-05-17T14:51:39.655 INFO:teuthology.orchestra.run.smithi195.stdout: osd journal size = 100 2024-05-17T14:51:39.655 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.655 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub load threshold = 5.0 2024-05-17T14:51:39.655 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub max interval = 600 2024-05-17T14:51:39.655 INFO:teuthology.orchestra.run.smithi195.stdout: osd mclock profile = high_recovery_ops 2024-05-17T14:51:39.656 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.656 INFO:teuthology.orchestra.run.smithi195.stdout: osd recover clone overlap = true 2024-05-17T14:51:39.656 INFO:teuthology.orchestra.run.smithi195.stdout: osd recovery max chunk = 1048576 2024-05-17T14:51:39.656 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.656 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug shutdown = true 2024-05-17T14:51:39.656 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug op order = true 2024-05-17T14:51:39.656 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify stray on activate = true 2024-05-17T14:51:39.656 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug trim objects = true 2024-05-17T14:51:39.656 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.656 INFO:teuthology.orchestra.run.smithi195.stdout: osd open classes on start = true 2024-05-17T14:51:39.656 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug pg log writeout = true 2024-05-17T14:51:39.657 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.657 INFO:teuthology.orchestra.run.smithi195.stdout: osd deep scrub update digest min age = 30 2024-05-17T14:51:39.657 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.657 INFO:teuthology.orchestra.run.smithi195.stdout: osd map max advance = 10 2024-05-17T14:51:39.657 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.657 INFO:teuthology.orchestra.run.smithi195.stdout: journal zero on create = true 2024-05-17T14:51:39.657 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.657 INFO:teuthology.orchestra.run.smithi195.stdout: filestore ondisk finisher threads = 3 2024-05-17T14:51:39.657 INFO:teuthology.orchestra.run.smithi195.stdout: filestore apply finisher threads = 3 2024-05-17T14:51:39.657 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.657 INFO:teuthology.orchestra.run.smithi195.stdout: bdev debug aio = true 2024-05-17T14:51:39.657 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug misdirected ops = true 2024-05-17T14:51:39.657 INFO:teuthology.orchestra.run.smithi195.stdout: bdev async discard = True 2024-05-17T14:51:39.657 INFO:teuthology.orchestra.run.smithi195.stdout: bdev enable discard = True 2024-05-17T14:51:39.657 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore block size = 96636764160 2024-05-17T14:51:39.658 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore fsck on mount = True 2024-05-17T14:51:39.658 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluefs = 1/20 2024-05-17T14:51:39.658 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluestore = 1/20 2024-05-17T14:51:39.658 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-05-17T14:51:39.658 INFO:teuthology.orchestra.run.smithi195.stdout: debug osd = 20 2024-05-17T14:51:39.658 INFO:teuthology.orchestra.run.smithi195.stdout: debug rocksdb = 4/10 2024-05-17T14:51:39.658 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd backfillfull_ratio = 0.85 2024-05-17T14:51:39.658 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd full ratio = 0.9 2024-05-17T14:51:39.658 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd nearfull ratio = 0.8 2024-05-17T14:51:39.658 INFO:teuthology.orchestra.run.smithi195.stdout: osd failsafe full ratio = 0.95 2024-05-17T14:51:39.658 INFO:teuthology.orchestra.run.smithi195.stdout: osd memory target = 1610612736 2024-05-17T14:51:39.658 INFO:teuthology.orchestra.run.smithi195.stdout: osd objectstore = bluestore 2024-05-17T14:51:39.658 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.659 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr] 2024-05-17T14:51:39.659 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-05-17T14:51:39.659 INFO:teuthology.orchestra.run.smithi195.stdout: debug mgr = 20 2024-05-17T14:51:39.659 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2024-05-17T14:51:39.659 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2024-05-17T14:51:39.659 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2024-05-17T14:51:39.659 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2024-05-17T14:51:39.659 INFO:teuthology.orchestra.run.smithi195.stdout: mgr/telemetry/nag = false 2024-05-17T14:51:39.659 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.659 INFO:teuthology.orchestra.run.smithi195.stdout:[mon] 2024-05-17T14:51:39.659 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-05-17T14:51:39.659 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2024-05-17T14:51:39.659 INFO:teuthology.orchestra.run.smithi195.stdout: debug paxos = 20 2024-05-17T14:51:39.659 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2024-05-17T14:51:39.660 INFO:teuthology.orchestra.run.smithi195.stdout: mon data avail warn = 5 2024-05-17T14:51:39.660 INFO:teuthology.orchestra.run.smithi195.stdout: mon mgr mkfs grace = 240 2024-05-17T14:51:39.660 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2024-05-17T14:51:39.660 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd reporter subtree level = osd 2024-05-17T14:51:39.660 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd prime pg temp = true 2024-05-17T14:51:39.660 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2024-05-17T14:51:39.660 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.660 INFO:teuthology.orchestra.run.smithi195.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-17T14:51:39.660 INFO:teuthology.orchestra.run.smithi195.stdout: auth mon ticket ttl = 660 # 11m 2024-05-17T14:51:39.660 INFO:teuthology.orchestra.run.smithi195.stdout: auth service ticket ttl = 240 # 4m 2024-05-17T14:51:39.660 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.660 INFO:teuthology.orchestra.run.smithi195.stdout: # don't complain about insecure global_id in the test suite 2024-05-17T14:51:39.660 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-17T14:51:39.661 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-17T14:51:39.661 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.661 INFO:teuthology.orchestra.run.smithi195.stdout: # 1m isn't quite enough 2024-05-17T14:51:39.661 INFO:teuthology.orchestra.run.smithi195.stdout: mon_down_mkfs_grace = 2m 2024-05-17T14:51:39.661 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.661 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_filestore_osds = false 2024-05-17T14:51:39.661 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:51:39.661 INFO:teuthology.orchestra.run.smithi195.stdout:[client] 2024-05-17T14:51:39.661 INFO:teuthology.orchestra.run.smithi195.stdout: rgw cache enabled = true 2024-05-17T14:51:39.661 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable ops log = true 2024-05-17T14:51:39.661 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable usage log = true 2024-05-17T14:51:39.661 INFO:teuthology.orchestra.run.smithi195.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-17T14:51:39.661 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-17T14:51:39.661 INFO:teuthology.orchestra.run.smithi195.stdout: debug rbd = 20 2024-05-17T14:51:39.662 INFO:teuthology.orchestra.run.smithi195.stdout: debug rbd_mirror = 15 2024-05-17T14:51:39.662 INFO:teuthology.orchestra.run.smithi195.stdout: log to stderr = False 2024-05-17T14:51:39.662 INFO:teuthology.orchestra.run.smithi195.stdout: rbd default features = 125 2024-05-17T14:51:39.662 INFO:teuthology.orchestra.run.smithi195.stdout: rbd mirror image policy type = simple 2024-05-17T14:51:39.662 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.a] 2024-05-17T14:51:39.662 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.0] 2024-05-17T14:51:39.662 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:51:39.662 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:51:39.662 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.1] 2024-05-17T14:51:39.662 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:51:39.662 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:51:39.662 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.2] 2024-05-17T14:51:39.662 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:51:39.662 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:51:39.663 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.3] 2024-05-17T14:51:39.663 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:51:39.663 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:51:39.664 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-05-17T14:51:39.664 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-05-17T14:51:39.757 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-17T14:51:39.757 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:39.758 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-05-17T14:51:39.773 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:39.773 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-05-17T14:51:39.829 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi134.front.sepia.ceph.com 2024-05-17T14:51:39.829 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:39.829 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/cluster1.keyring 2024-05-17T14:51:39.829 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-05-17T14:51:39.903 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:39.903 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-05-17T14:51:39.958 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi195.front.sepia.ceph.com 2024-05-17T14:51:39.958 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:51:39.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/cluster1.keyring 2024-05-17T14:51:39.958 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-05-17T14:51:40.000 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:51:40.000 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-05-17T14:51:40.058 INFO:tasks.ceph:Setting up mon nodes... 2024-05-17T14:51:40.058 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-17T14:51:40.059 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-05-17T14:51:40.110 INFO:teuthology.orchestra.run.smithi134.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-05-17T14:51:40.112 INFO:tasks.ceph:Setting up mds nodes... 2024-05-17T14:51:40.113 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-17T14:51:40.113 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-05-17T14:51:40.153 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-05-17T14:51:40.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-05-17T14:51:40.225 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-05-17T14:51:40.238 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-05-17T14:51:40.280 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-05-17T14:51:40.293 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-05-17T14:51:40.374 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-05-17T14:51:40.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-05-17T14:51:40.467 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-05-17T14:51:40.480 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-05-17T14:51:40.561 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-05-17T14:51:40.573 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-05-17T14:51:40.654 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-05-17T14:51:40.666 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-05-17T14:51:40.747 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-05-17T14:51:40.759 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-05-17T14:51:40.842 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-05-17T14:51:40.852 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-17T14:51:40.852 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-05-17T14:51:40.852 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-05-17T14:51:40.877 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-17T14:51:40.877 INFO:tasks.ceph:role: cluster1.osd.0 2024-05-17T14:51:40.877 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com 2024-05-17T14:51:40.877 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-17T14:51:40.949 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-17T14:51:40.949 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-17T14:51:40.949 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-17T14:51:40.949 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-17T14:51:40.950 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-17T14:51:40.950 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-05-17T14:51:40.950 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-17T14:51:40.950 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-17T14:51:40.950 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-17T14:51:40.950 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-17T14:51:41.443 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-05-17T14:51:41.445 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-05-17T14:51:41.445 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-05-17T14:51:41.577 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-05-17T14:51:41.611 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-05-17T14:51:41.676 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-17T14:51:41.677 INFO:tasks.ceph:role: cluster1.osd.1 2024-05-17T14:51:41.677 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com 2024-05-17T14:51:41.677 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-17T14:51:41.748 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-17T14:51:41.748 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-17T14:51:41.748 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-17T14:51:41.749 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-17T14:51:41.749 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-17T14:51:41.749 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-05-17T14:51:41.749 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-17T14:51:41.749 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-17T14:51:41.749 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-17T14:51:41.749 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-17T14:51:42.202 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-05-17T14:51:42.204 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-05-17T14:51:42.205 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-05-17T14:51:42.304 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-05-17T14:51:42.333 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-05-17T14:51:42.398 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-17T14:51:42.398 INFO:tasks.ceph:role: cluster1.osd.2 2024-05-17T14:51:42.398 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi134.front.sepia.ceph.com 2024-05-17T14:51:42.399 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-17T14:51:42.470 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-17T14:51:42.470 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-17T14:51:42.470 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-17T14:51:42.470 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-17T14:51:42.470 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-17T14:51:42.470 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-05-17T14:51:42.470 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-17T14:51:42.470 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-17T14:51:42.471 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-17T14:51:42.471 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-17T14:51:42.953 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-05-17T14:51:42.955 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-05-17T14:51:42.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-05-17T14:51:43.107 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-05-17T14:51:43.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-05-17T14:51:43.234 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:43.233+0000 7f033c419640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-05-17T14:51:43.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:43.234+0000 7f033c419640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-05-17T14:51:43.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:43.234+0000 7f033c419640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-05-17T14:51:43.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:43.234+0000 7f033c419640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-05-17T14:51:43.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:43.234+0000 7f033c419640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-05-17T14:51:43.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:43.242+0000 7f033c419640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-05-17T14:51:46.094 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-05-17T14:51:46.124 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-05-17T14:51:46.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:46.214+0000 7fdbac211640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-05-17T14:51:46.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:46.214+0000 7fdbac211640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-05-17T14:51:46.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:46.214+0000 7fdbac211640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-05-17T14:51:46.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:46.214+0000 7fdbac211640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-05-17T14:51:46.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:46.214+0000 7fdbac211640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-05-17T14:51:46.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:46.215+0000 7fdbac211640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-05-17T14:51:49.061 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-05-17T14:51:49.093 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-05-17T14:51:49.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:49.184+0000 7fc1c069a640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-05-17T14:51:49.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:49.184+0000 7fc1c069a640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-05-17T14:51:49.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:49.184+0000 7fc1c069a640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-05-17T14:51:49.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:49.184+0000 7fc1c069a640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-05-17T14:51:49.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:49.184+0000 7fc1c069a640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-05-17T14:51:49.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-05-17T14:51:49.185+0000 7fc1c069a640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-05-17T14:51:52.045 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-05-17T14:51:52.075 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-17T14:51:52.075 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:52.075 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-05-17T14:51:52.143 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:52.143 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-05-17T14:51:52.211 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:52.211 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-05-17T14:51:52.278 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:52.278 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-05-17T14:51:52.345 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:52.345 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-05-17T14:51:52.401 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:51:52.401 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-05-17T14:51:52.421 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:51:52.421 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-05-17T14:51:52.478 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:51:52.478 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-05-17T14:51:52.536 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:51:52.536 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-05-17T14:51:52.592 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:51:52.592 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-05-17T14:51:52.650 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:51:52.650 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-05-17T14:51:52.707 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:51:52.707 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-05-17T14:51:52.764 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:51:52.764 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-05-17T14:51:52.821 INFO:tasks.ceph:Adding keys to all mons... 2024-05-17T14:51:52.821 DEBUG:teuthology.orchestra.run.smithi134:> sudo tee -a /etc/ceph/cluster1.keyring 2024-05-17T14:51:52.849 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr.x] 2024-05-17T14:51:52.849 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD8bkdmn9SaBhAADPLg3HxQyiby/IQz7GJodQ== 2024-05-17T14:51:52.849 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.0] 2024-05-17T14:51:52.849 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD/bkdmJRwADhAA9GvjlbIk86c25kTlr15U5Q== 2024-05-17T14:51:52.849 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.1] 2024-05-17T14:51:52.849 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQACb0dmBF7YDBAAZgZYQkEtLYcOFKyVJ87Wtg== 2024-05-17T14:51:52.849 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.2] 2024-05-17T14:51:52.849 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQAFb0dm2CYNCxAAu6rc9/rvdMypUalavikqMw== 2024-05-17T14:51:52.849 INFO:teuthology.orchestra.run.smithi134.stdout:[client.0] 2024-05-17T14:51:52.850 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD8bkdmaWAxCRAA3Wu36M04rFseXfwIOe80jw== 2024-05-17T14:51:52.850 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror] 2024-05-17T14:51:52.850 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD8bkdmmmZ0DRAAmOKW3JTaaphl9Im2vH3/3w== 2024-05-17T14:51:52.850 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.0] 2024-05-17T14:51:52.850 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD8bkdmx0bAEBAAidcTnG2fI9jQHMCLDCGlGA== 2024-05-17T14:51:52.850 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.1] 2024-05-17T14:51:52.850 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD8bkdm8y1WFhAApbXIdTLv6kZwRjvH37Yx7w== 2024-05-17T14:51:52.850 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.2] 2024-05-17T14:51:52.850 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD8bkdmkwbqGxAAnd6web7SHzC54zR+7AI8OA== 2024-05-17T14:51:52.850 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.3] 2024-05-17T14:51:52.850 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD8bkdmbCt7IRAAYztVN4zwiqXoBJUEnAmM8A== 2024-05-17T14:51:52.850 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.4] 2024-05-17T14:51:52.850 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD8bkdmFWUHJxAANDeYXaz4DAXf3eiFj6t2Uw== 2024-05-17T14:51:52.851 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.5] 2024-05-17T14:51:52.851 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD8bkdmDXiQLBAA13ftvjTmxnFNcHIYWKOprA== 2024-05-17T14:51:52.851 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.6] 2024-05-17T14:51:52.851 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQD8bkdmAk0WMhAA01zJ+NMFXTk3YzMGpIa1pQ== 2024-05-17T14:51:52.852 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-17T14:51:52.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-17T14:51:53.000 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-17T14:51:53.052 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-17T14:51:53.102 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:51:53.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:51:53.208 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:51:53.261 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:51:53.315 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:51:53.368 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:51:53.421 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:51:53.475 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:51:53.528 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:51:53.581 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-17T14:51:53.582 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-05-17T14:51:53.606 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-05-17T14:51:53.702 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-05-17T14:51:53.726 DEBUG:teuthology.orchestra.run.smithi134:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-05-17T14:51:53.782 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-05-17T14:51:53.782 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-05-17T14:51:53.782 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-05-17T14:51:53.824 INFO:tasks.ceph.cluster1.mon.a:Started 2024-05-17T14:51:53.824 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-05-17T14:51:53.824 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-05-17T14:51:53.824 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-05-17T14:51:53.826 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-05-17T14:51:53.826 INFO:tasks.ceph:Setting crush tunables to default 2024-05-17T14:51:53.827 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster1 osd crush tunables default 2024-05-17T14:51:54.002 INFO:teuthology.orchestra.run.smithi134.stderr:adjusted tunables profile to default 2024-05-17T14:51:54.013 INFO:tasks.ceph:check_enable_crimson: False 2024-05-17T14:51:54.013 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-05-17T14:51:54.013 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:54.013 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-05-17T14:51:54.038 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:54.038 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-05-17T14:51:54.104 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:51:54.104 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-05-17T14:51:54.170 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster1 osd new 8cfff074-c907-4e95-9f1c-6a2b49a58a0b 0 2024-05-17T14:51:54.209 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:54.208+0000 7fd16dc13040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-17T14:51:54.367 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:54.366+0000 7fd16dc13040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-17T14:51:54.389 INFO:teuthology.orchestra.run.smithi134.stdout:0 2024-05-17T14:51:54.402 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster1 osd new 5c868c7d-1952-47d5-b42c-dc2b38e77e72 1 2024-05-17T14:51:54.424 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:54.423+0000 7fd16dc13040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-17T14:51:54.583 INFO:teuthology.orchestra.run.smithi134.stdout:1 2024-05-17T14:51:54.595 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster1 osd new a9106acf-6f85-401d-b478-d79dd44d927a 2 2024-05-17T14:51:54.694 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:54.693+0000 7fd16dc13040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-17T14:51:54.742 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:54.742+0000 7fd16dc13040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-17T14:51:54.817 INFO:teuthology.orchestra.run.smithi134.stdout:2 2024-05-17T14:51:54.828 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-05-17T14:51:54.829 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-05-17T14:51:54.831 INFO:tasks.ceph.cluster1.osd.0:Started 2024-05-17T14:51:54.831 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-05-17T14:51:54.831 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-05-17T14:51:54.833 INFO:tasks.ceph.cluster1.osd.1:Started 2024-05-17T14:51:54.833 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-05-17T14:51:54.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-05-17T14:51:54.836 INFO:tasks.ceph.cluster1.osd.2:Started 2024-05-17T14:51:54.837 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-05-17T14:51:55.080 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:51:55.080 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":5,"fsid":"5b29e51e-670a-453e-8d22-58d6f74d2947","created":"2024-05-17T14:51:53.910460+0000","modified":"2024-05-17T14:51:54.817025+0000","last_up_change":"0.000000","last_in_change":"2024-05-17T14:51:54.817025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8cfff074-c907-4e95-9f1c-6a2b49a58a0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c868c7d-1952-47d5-b42c-dc2b38e77e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a9106acf-6f85-401d-b478-d79dd44d927a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-17T14:51:55.091 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-05-17T14:51:55.091 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-17T14:51:55.394 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-05-17T14:51:55.433 INFO:tasks.ceph.cluster1.osd.0.smithi134.stderr:2024-05-17T14:51:55.432+0000 7fd85e60c640 -1 Falling back to public interface 2024-05-17T14:51:55.436 INFO:tasks.ceph.cluster1.osd.2.smithi134.stderr:2024-05-17T14:51:55.436+0000 7f4370a0c640 -1 Falling back to public interface 2024-05-17T14:51:55.440 INFO:tasks.ceph.cluster1.osd.1.smithi134.stderr:2024-05-17T14:51:55.439+0000 7f346e1b2640 -1 Falling back to public interface 2024-05-17T14:51:55.557 INFO:teuthology.misc.health.smithi134.stdout: 2024-05-17T14:51:55.557 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":5,"fsid":"5b29e51e-670a-453e-8d22-58d6f74d2947","created":"2024-05-17T14:51:53.910460+0000","modified":"2024-05-17T14:51:54.817025+0000","last_up_change":"0.000000","last_in_change":"2024-05-17T14:51:54.817025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8cfff074-c907-4e95-9f1c-6a2b49a58a0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c868c7d-1952-47d5-b42c-dc2b38e77e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a9106acf-6f85-401d-b478-d79dd44d927a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-17T14:51:55.567 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-05-17T14:51:55.583 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:55.582+0000 7fd16dc13040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-17T14:51:55.670 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:55.669+0000 7fd16dc13040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-17T14:51:55.713 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:55.712+0000 7fd16dc13040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-17T14:51:55.890 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:55.889+0000 7fd16dc13040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-17T14:51:55.938 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:55.937+0000 7fd16dc13040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-17T14:51:55.992 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:55.991+0000 7fd16dc13040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-17T14:51:56.106 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:56.105+0000 7fd16dc13040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-17T14:51:56.483 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:56.482+0000 7fd16dc13040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-17T14:51:56.622 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:56.621+0000 7fd16dc13040 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-17T14:51:56.881 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:56.881+0000 7fd16dc13040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-17T14:51:56.933 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:56.933+0000 7fd16dc13040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-17T14:51:57.049 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:57.049+0000 7fd16dc13040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-17T14:51:57.337 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:57.336+0000 7fd16dc13040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-17T14:51:57.386 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:57.386+0000 7fd16dc13040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-17T14:51:57.542 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:57.541+0000 7fd16dc13040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-17T14:51:57.680 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:57.679+0000 7fd16dc13040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-17T14:51:57.729 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:57.728+0000 7fd16dc13040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-17T14:51:57.799 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:57.798+0000 7fd16dc13040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-17T14:51:57.844 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:57.843+0000 7fd16dc13040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-17T14:51:58.343 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:58.342+0000 7fd16dc13040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-17T14:51:58.458 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:58.457+0000 7fd16dc13040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-17T14:51:58.944 INFO:tasks.ceph.cluster1.osd.1.smithi134.stderr:2024-05-17T14:51:58.944+0000 7f346e1b2640 -1 osd.1 0 log_to_monitors true 2024-05-17T14:51:58.983 INFO:tasks.ceph.cluster1.osd.0.smithi134.stderr:2024-05-17T14:51:58.983+0000 7fd85e60c640 -1 osd.0 0 log_to_monitors true 2024-05-17T14:51:58.990 INFO:tasks.ceph.cluster1.osd.2.smithi134.stderr:2024-05-17T14:51:58.989+0000 7f4370a0c640 -1 osd.2 0 log_to_monitors true 2024-05-17T14:51:59.466 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T14:51:59.465+0000 7fd10d536640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-05-17T14:52:01.870 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-05-17T14:52:02.248 INFO:tasks.ceph.cluster1.osd.0.smithi134.stderr:2024-05-17T14:52:02.247+0000 7fd85a075640 -1 osd.0 0 waiting for initial osdmap 2024-05-17T14:52:02.267 INFO:tasks.ceph.cluster1.osd.0.smithi134.stderr:2024-05-17T14:52:02.267+0000 7fd851e95640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-17T14:52:02.286 INFO:tasks.ceph.cluster1.osd.1.smithi134.stderr:2024-05-17T14:52:02.285+0000 7f346a95c640 -1 osd.1 0 waiting for initial osdmap 2024-05-17T14:52:02.297 INFO:tasks.ceph.cluster1.osd.2.smithi134.stderr:2024-05-17T14:52:02.296+0000 7f436c3df640 -1 osd.2 0 waiting for initial osdmap 2024-05-17T14:52:02.306 INFO:tasks.ceph.cluster1.osd.1.smithi134.stderr:2024-05-17T14:52:02.306+0000 7f3461f68640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-17T14:52:02.310 INFO:teuthology.misc.health.smithi134.stdout: 2024-05-17T14:52:02.310 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":7,"fsid":"5b29e51e-670a-453e-8d22-58d6f74d2947","created":"2024-05-17T14:51:53.910460+0000","modified":"2024-05-17T14:52:00.516385+0000","last_up_change":"0.000000","last_in_change":"2024-05-17T14:51:54.817025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8cfff074-c907-4e95-9f1c-6a2b49a58a0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5c868c7d-1952-47d5-b42c-dc2b38e77e72","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a9106acf-6f85-401d-b478-d79dd44d927a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-17T14:52:02.319 INFO:tasks.ceph.cluster1.osd.2.smithi134.stderr:2024-05-17T14:52:02.319+0000 7f43641ff640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-17T14:52:02.322 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-05-17T14:52:08.625 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-05-17T14:52:08.922 INFO:teuthology.misc.health.smithi134.stdout: 2024-05-17T14:52:08.922 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":11,"fsid":"5b29e51e-670a-453e-8d22-58d6f74d2947","created":"2024-05-17T14:51:53.910460+0000","modified":"2024-05-17T14:52:06.539695+0000","last_up_change":"2024-05-17T14:52:03.249435+0000","last_in_change":"2024-05-17T14:51:54.817025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-17T14:52:04.480981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfff074-c907-4e95-9f1c-6a2b49a58a0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3627481211},{"type":"v1","addr":"172.21.15.134:6801","nonce":3627481211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3627481211},{"type":"v1","addr":"172.21.15.134:6803","nonce":3627481211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3627481211},{"type":"v1","addr":"172.21.15.134:6807","nonce":3627481211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3627481211},{"type":"v1","addr":"172.21.15.134:6805","nonce":3627481211}]},"public_addr":"172.21.15.134:6801/3627481211","cluster_addr":"172.21.15.134:6803/3627481211","heartbeat_back_addr":"172.21.15.134:6807/3627481211","heartbeat_front_addr":"172.21.15.134:6805/3627481211","state":["exists","up"]},{"osd":1,"uuid":"5c868c7d-1952-47d5-b42c-dc2b38e77e72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2915684477},{"type":"v1","addr":"172.21.15.134:6817","nonce":2915684477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2915684477},{"type":"v1","addr":"172.21.15.134:6819","nonce":2915684477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2915684477},{"type":"v1","addr":"172.21.15.134:6823","nonce":2915684477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2915684477},{"type":"v1","addr":"172.21.15.134:6821","nonce":2915684477}]},"public_addr":"172.21.15.134:6817/2915684477","cluster_addr":"172.21.15.134:6819/2915684477","heartbeat_back_addr":"172.21.15.134:6823/2915684477","heartbeat_front_addr":"172.21.15.134:6821/2915684477","state":["exists","up"]},{"osd":2,"uuid":"a9106acf-6f85-401d-b478-d79dd44d927a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":579299408},{"type":"v1","addr":"172.21.15.134:6809","nonce":579299408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":579299408},{"type":"v1","addr":"172.21.15.134:6811","nonce":579299408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":579299408},{"type":"v1","addr":"172.21.15.134:6815","nonce":579299408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":579299408},{"type":"v1","addr":"172.21.15.134:6813","nonce":579299408}]},"public_addr":"172.21.15.134:6809/579299408","cluster_addr":"172.21.15.134:6811/579299408","heartbeat_back_addr":"172.21.15.134:6815/579299408","heartbeat_front_addr":"172.21.15.134:6813/579299408","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-17T14:51:59.980216+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-17T14:51:59.990041+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-17T14:52:00.023148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-17T14:52:08.934 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-05-17T14:52:08.934 INFO:tasks.ceph:Creating RBD pool 2024-05-17T14:52:08.934 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-05-17T14:52:09.553 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'rbd' created 2024-05-17T14:52:09.568 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-05-17T14:52:10.556 INFO:teuthology.orchestra.run.smithi134.stderr:enabled application 'rbd' on pool 'rbd' 2024-05-17T14:52:10.576 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-05-17T14:52:10.576 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-05-17T14:52:10.577 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-05-17T14:52:10.577 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-05-17T14:52:10.577 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-17T14:52:10.922 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:10.933 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":1963551219},{"type":"v1","addr":"172.21.15.134:6825","nonce":1963551219}]},"active_addr":"172.21.15.134:6825/1963551219","active_change":"2024-05-17T14:51:58.460133+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":984243436}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":1488325654}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":4094553668}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":4005272245}]}]} 2024-05-17T14:52:10.937 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-05-17T14:52:10.937 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-05-17T14:52:10.937 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-05-17T14:52:11.227 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:11.228 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":13,"fsid":"5b29e51e-670a-453e-8d22-58d6f74d2947","created":"2024-05-17T14:51:53.910460+0000","modified":"2024-05-17T14:52:10.554425+0000","last_up_change":"2024-05-17T14:52:03.249435+0000","last_in_change":"2024-05-17T14:51:54.817025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-17T14:52:04.480981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-17T14:52:09.227358+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfff074-c907-4e95-9f1c-6a2b49a58a0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3627481211},{"type":"v1","addr":"172.21.15.134:6801","nonce":3627481211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3627481211},{"type":"v1","addr":"172.21.15.134:6803","nonce":3627481211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3627481211},{"type":"v1","addr":"172.21.15.134:6807","nonce":3627481211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3627481211},{"type":"v1","addr":"172.21.15.134:6805","nonce":3627481211}]},"public_addr":"172.21.15.134:6801/3627481211","cluster_addr":"172.21.15.134:6803/3627481211","heartbeat_back_addr":"172.21.15.134:6807/3627481211","heartbeat_front_addr":"172.21.15.134:6805/3627481211","state":["exists","up"]},{"osd":1,"uuid":"5c868c7d-1952-47d5-b42c-dc2b38e77e72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2915684477},{"type":"v1","addr":"172.21.15.134:6817","nonce":2915684477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2915684477},{"type":"v1","addr":"172.21.15.134:6819","nonce":2915684477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2915684477},{"type":"v1","addr":"172.21.15.134:6823","nonce":2915684477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2915684477},{"type":"v1","addr":"172.21.15.134:6821","nonce":2915684477}]},"public_addr":"172.21.15.134:6817/2915684477","cluster_addr":"172.21.15.134:6819/2915684477","heartbeat_back_addr":"172.21.15.134:6823/2915684477","heartbeat_front_addr":"172.21.15.134:6821/2915684477","state":["exists","up"]},{"osd":2,"uuid":"a9106acf-6f85-401d-b478-d79dd44d927a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":579299408},{"type":"v1","addr":"172.21.15.134:6809","nonce":579299408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":579299408},{"type":"v1","addr":"172.21.15.134:6811","nonce":579299408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":579299408},{"type":"v1","addr":"172.21.15.134:6815","nonce":579299408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":579299408},{"type":"v1","addr":"172.21.15.134:6813","nonce":579299408}]},"public_addr":"172.21.15.134:6809/579299408","cluster_addr":"172.21.15.134:6811/579299408","heartbeat_back_addr":"172.21.15.134:6815/579299408","heartbeat_front_addr":"172.21.15.134:6813/579299408","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-17T14:51:59.980216+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-17T14:51:59.990041+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-17T14:52:00.023148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-17T14:52:11.240 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-05-17T14:52:11.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-05-17T14:52:11.531 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:11.531 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":13,"fsid":"5b29e51e-670a-453e-8d22-58d6f74d2947","created":"2024-05-17T14:51:53.910460+0000","modified":"2024-05-17T14:52:10.554425+0000","last_up_change":"2024-05-17T14:52:03.249435+0000","last_in_change":"2024-05-17T14:51:54.817025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-17T14:52:04.480981+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-17T14:52:09.227358+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8cfff074-c907-4e95-9f1c-6a2b49a58a0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3627481211},{"type":"v1","addr":"172.21.15.134:6801","nonce":3627481211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3627481211},{"type":"v1","addr":"172.21.15.134:6803","nonce":3627481211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3627481211},{"type":"v1","addr":"172.21.15.134:6807","nonce":3627481211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3627481211},{"type":"v1","addr":"172.21.15.134:6805","nonce":3627481211}]},"public_addr":"172.21.15.134:6801/3627481211","cluster_addr":"172.21.15.134:6803/3627481211","heartbeat_back_addr":"172.21.15.134:6807/3627481211","heartbeat_front_addr":"172.21.15.134:6805/3627481211","state":["exists","up"]},{"osd":1,"uuid":"5c868c7d-1952-47d5-b42c-dc2b38e77e72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2915684477},{"type":"v1","addr":"172.21.15.134:6817","nonce":2915684477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2915684477},{"type":"v1","addr":"172.21.15.134:6819","nonce":2915684477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2915684477},{"type":"v1","addr":"172.21.15.134:6823","nonce":2915684477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2915684477},{"type":"v1","addr":"172.21.15.134:6821","nonce":2915684477}]},"public_addr":"172.21.15.134:6817/2915684477","cluster_addr":"172.21.15.134:6819/2915684477","heartbeat_back_addr":"172.21.15.134:6823/2915684477","heartbeat_front_addr":"172.21.15.134:6821/2915684477","state":["exists","up"]},{"osd":2,"uuid":"a9106acf-6f85-401d-b478-d79dd44d927a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":579299408},{"type":"v1","addr":"172.21.15.134:6809","nonce":579299408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":579299408},{"type":"v1","addr":"172.21.15.134:6811","nonce":579299408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":579299408},{"type":"v1","addr":"172.21.15.134:6815","nonce":579299408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":579299408},{"type":"v1","addr":"172.21.15.134:6813","nonce":579299408}]},"public_addr":"172.21.15.134:6809/579299408","cluster_addr":"172.21.15.134:6811/579299408","heartbeat_back_addr":"172.21.15.134:6815/579299408","heartbeat_front_addr":"172.21.15.134:6813/579299408","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-17T14:51:59.980216+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-17T14:51:59.990041+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-17T14:52:00.023148+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-17T14:52:11.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-05-17T14:52:11.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-05-17T14:52:11.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-05-17T14:52:11.688 INFO:teuthology.orchestra.run.smithi134.stdout:34359738373 2024-05-17T14:52:11.688 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-05-17T14:52:11.692 INFO:teuthology.orchestra.run.smithi134.stdout:34359738373 2024-05-17T14:52:11.693 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-05-17T14:52:11.693 INFO:teuthology.orchestra.run.smithi134.stdout:34359738374 2024-05-17T14:52:11.694 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-05-17T14:52:11.997 INFO:teuthology.orchestra.run.smithi134.stdout:34359738372 2024-05-17T14:52:12.010 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.2 2024-05-17T14:52:12.059 INFO:teuthology.orchestra.run.smithi134.stdout:34359738373 2024-05-17T14:52:12.067 INFO:teuthology.orchestra.run.smithi134.stdout:34359738372 2024-05-17T14:52:12.072 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738373 for osd.1 2024-05-17T14:52:12.080 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.0 2024-05-17T14:52:13.011 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-05-17T14:52:13.074 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-05-17T14:52:13.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-05-17T14:52:13.334 INFO:teuthology.orchestra.run.smithi134.stdout:34359738373 2024-05-17T14:52:13.347 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.2 2024-05-17T14:52:13.348 DEBUG:teuthology.parallel:result is None 2024-05-17T14:52:13.389 INFO:teuthology.orchestra.run.smithi134.stdout:34359738374 2024-05-17T14:52:13.402 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.1 2024-05-17T14:52:13.403 DEBUG:teuthology.parallel:result is None 2024-05-17T14:52:13.430 INFO:teuthology.orchestra.run.smithi134.stdout:34359738373 2024-05-17T14:52:13.443 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.0 2024-05-17T14:52:13.443 DEBUG:teuthology.parallel:result is None 2024-05-17T14:52:13.443 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-05-17T14:52:13.443 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-05-17T14:52:13.774 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:13.775 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-05-17T14:52:13.787 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-05-17T14:52:12.465983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82556,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032964,"statfs":{"total":289910292480,"available":289825755136,"internally_reserved":0,"allocated":1531904,"data_stored":1274801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.911603"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:10.570562+0000","last_change":"2024-05-17T14:52:10.567166+0000","last_active":"2024-05-17T14:52:10.570562+0000","last_peered":"2024-05-17T14:52:10.570562+0000","last_clean":"2024-05-17T14:52:10.570562+0000","last_became_active":"2024-05-17T14:52:10.566842+0000","last_became_peered":"2024-05-17T14:52:10.566842+0000","last_unstale":"2024-05-17T14:52:10.570562+0000","last_undegraded":"2024-05-17T14:52:10.570562+0000","last_fullsized":"2024-05-17T14:52:10.570562+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_clean_scrub_stamp":"2024-05-17T14:52:09.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T23:47:09.444169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:10.568965+0000","last_change":"2024-05-17T14:52:10.564960+0000","last_active":"2024-05-17T14:52:10.568965+0000","last_peered":"2024-05-17T14:52:10.568965+0000","last_clean":"2024-05-17T14:52:10.568965+0000","last_became_active":"2024-05-17T14:52:10.564415+0000","last_became_peered":"2024-05-17T14:52:10.564415+0000","last_unstale":"2024-05-17T14:52:10.568965+0000","last_undegraded":"2024-05-17T14:52:10.568965+0000","last_fullsized":"2024-05-17T14:52:10.568965+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_clean_scrub_stamp":"2024-05-17T14:52:09.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-19T00:23:04.842075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:10.570587+0000","last_change":"2024-05-17T14:52:10.567209+0000","last_active":"2024-05-17T14:52:10.570587+0000","last_peered":"2024-05-17T14:52:10.570587+0000","last_clean":"2024-05-17T14:52:10.570587+0000","last_became_active":"2024-05-17T14:52:10.566899+0000","last_became_peered":"2024-05-17T14:52:10.566899+0000","last_unstale":"2024-05-17T14:52:10.570587+0000","last_undegraded":"2024-05-17T14:52:10.570587+0000","last_fullsized":"2024-05-17T14:52:10.570587+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_clean_scrub_stamp":"2024-05-17T14:52:09.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T15:25:56.981540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:10.570570+0000","last_change":"2024-05-17T14:52:10.567203+0000","last_active":"2024-05-17T14:52:10.570570+0000","last_peered":"2024-05-17T14:52:10.570570+0000","last_clean":"2024-05-17T14:52:10.570570+0000","last_became_active":"2024-05-17T14:52:10.566889+0000","last_became_peered":"2024-05-17T14:52:10.566889+0000","last_unstale":"2024-05-17T14:52:10.570570+0000","last_undegraded":"2024-05-17T14:52:10.570570+0000","last_fullsized":"2024-05-17T14:52:10.570570+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_clean_scrub_stamp":"2024-05-17T14:52:09.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T23:55:57.213083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-17T14:52:11.558984+0000","last_change":"2024-05-17T14:52:10.564929+0000","last_active":"2024-05-17T14:52:11.558984+0000","last_peered":"2024-05-17T14:52:11.558984+0000","last_clean":"2024-05-17T14:52:11.558984+0000","last_became_active":"2024-05-17T14:52:10.564625+0000","last_became_peered":"2024-05-17T14:52:10.564625+0000","last_unstale":"2024-05-17T14:52:11.558984+0000","last_undegraded":"2024-05-17T14:52:11.558984+0000","last_fullsized":"2024-05-17T14:52:11.558984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_clean_scrub_stamp":"2024-05-17T14:52:09.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T15:48:04.048372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:10.570447+0000","last_change":"2024-05-17T14:52:10.567903+0000","last_active":"2024-05-17T14:52:10.570447+0000","last_peered":"2024-05-17T14:52:10.570447+0000","last_clean":"2024-05-17T14:52:10.570447+0000","last_became_active":"2024-05-17T14:52:10.567621+0000","last_became_peered":"2024-05-17T14:52:10.567621+0000","last_unstale":"2024-05-17T14:52:10.570447+0000","last_undegraded":"2024-05-17T14:52:10.570447+0000","last_fullsized":"2024-05-17T14:52:10.570447+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_clean_scrub_stamp":"2024-05-17T14:52:09.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-19T00:15:36.563657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:10.570401+0000","last_change":"2024-05-17T14:52:10.567390+0000","last_active":"2024-05-17T14:52:10.570401+0000","last_peered":"2024-05-17T14:52:10.570401+0000","last_clean":"2024-05-17T14:52:10.570401+0000","last_became_active":"2024-05-17T14:52:10.567087+0000","last_became_peered":"2024-05-17T14:52:10.567087+0000","last_unstale":"2024-05-17T14:52:10.570401+0000","last_undegraded":"2024-05-17T14:52:10.570401+0000","last_fullsized":"2024-05-17T14:52:10.570401+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_clean_scrub_stamp":"2024-05-17T14:52:09.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-19T01:42:59.573342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:10.568961+0000","last_change":"2024-05-17T14:52:10.564672+0000","last_active":"2024-05-17T14:52:10.568961+0000","last_peered":"2024-05-17T14:52:10.568961+0000","last_clean":"2024-05-17T14:52:10.568961+0000","last_became_active":"2024-05-17T14:52:10.564380+0000","last_became_peered":"2024-05-17T14:52:10.564380+0000","last_unstale":"2024-05-17T14:52:10.568961+0000","last_undegraded":"2024-05-17T14:52:10.568961+0000","last_fullsized":"2024-05-17T14:52:10.568961+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_clean_scrub_stamp":"2024-05-17T14:52:09.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T22:46:21.935245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":83,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:10.561959+0000","last_change":"2024-05-17T14:52:06.027203+0000","last_active":"2024-05-17T14:52:10.561959+0000","last_peered":"2024-05-17T14:52:10.561959+0000","last_clean":"2024-05-17T14:52:10.561959+0000","last_became_active":"2024-05-17T14:52:06.026327+0000","last_became_peered":"2024-05-17T14:52:06.026327+0000","last_unstale":"2024-05-17T14:52:10.561959+0000","last_undegraded":"2024-05-17T14:52:10.561959+0000","last_fullsized":"2024-05-17T14:52:10.561959+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:04.530871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:04.530871+0000","last_clean_scrub_stamp":"2024-05-17T14:52:04.530871+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T16:53:15.484878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-17T14:52:13.788 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-05-17T14:52:14.079 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:14.080 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-05-17T14:52:14.092 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-05-17T14:52:12.465983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82556,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032964,"statfs":{"total":289910292480,"available":289825755136,"internally_reserved":0,"allocated":1531904,"data_stored":1274801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.911603"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:10.570562+0000","last_change":"2024-05-17T14:52:10.567166+0000","last_active":"2024-05-17T14:52:10.570562+0000","last_peered":"2024-05-17T14:52:10.570562+0000","last_clean":"2024-05-17T14:52:10.570562+0000","last_became_active":"2024-05-17T14:52:10.566842+0000","last_became_peered":"2024-05-17T14:52:10.566842+0000","last_unstale":"2024-05-17T14:52:10.570562+0000","last_undegraded":"2024-05-17T14:52:10.570562+0000","last_fullsized":"2024-05-17T14:52:10.570562+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_clean_scrub_stamp":"2024-05-17T14:52:09.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T23:47:09.444169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:10.568965+0000","last_change":"2024-05-17T14:52:10.564960+0000","last_active":"2024-05-17T14:52:10.568965+0000","last_peered":"2024-05-17T14:52:10.568965+0000","last_clean":"2024-05-17T14:52:10.568965+0000","last_became_active":"2024-05-17T14:52:10.564415+0000","last_became_peered":"2024-05-17T14:52:10.564415+0000","last_unstale":"2024-05-17T14:52:10.568965+0000","last_undegraded":"2024-05-17T14:52:10.568965+0000","last_fullsized":"2024-05-17T14:52:10.568965+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_clean_scrub_stamp":"2024-05-17T14:52:09.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-19T00:23:04.842075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:10.570587+0000","last_change":"2024-05-17T14:52:10.567209+0000","last_active":"2024-05-17T14:52:10.570587+0000","last_peered":"2024-05-17T14:52:10.570587+0000","last_clean":"2024-05-17T14:52:10.570587+0000","last_became_active":"2024-05-17T14:52:10.566899+0000","last_became_peered":"2024-05-17T14:52:10.566899+0000","last_unstale":"2024-05-17T14:52:10.570587+0000","last_undegraded":"2024-05-17T14:52:10.570587+0000","last_fullsized":"2024-05-17T14:52:10.570587+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_clean_scrub_stamp":"2024-05-17T14:52:09.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T15:25:56.981540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:10.570570+0000","last_change":"2024-05-17T14:52:10.567203+0000","last_active":"2024-05-17T14:52:10.570570+0000","last_peered":"2024-05-17T14:52:10.570570+0000","last_clean":"2024-05-17T14:52:10.570570+0000","last_became_active":"2024-05-17T14:52:10.566889+0000","last_became_peered":"2024-05-17T14:52:10.566889+0000","last_unstale":"2024-05-17T14:52:10.570570+0000","last_undegraded":"2024-05-17T14:52:10.570570+0000","last_fullsized":"2024-05-17T14:52:10.570570+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_clean_scrub_stamp":"2024-05-17T14:52:09.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T23:55:57.213083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-05-17T14:52:11.558984+0000","last_change":"2024-05-17T14:52:10.564929+0000","last_active":"2024-05-17T14:52:11.558984+0000","last_peered":"2024-05-17T14:52:11.558984+0000","last_clean":"2024-05-17T14:52:11.558984+0000","last_became_active":"2024-05-17T14:52:10.564625+0000","last_became_peered":"2024-05-17T14:52:10.564625+0000","last_unstale":"2024-05-17T14:52:11.558984+0000","last_undegraded":"2024-05-17T14:52:11.558984+0000","last_fullsized":"2024-05-17T14:52:11.558984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_clean_scrub_stamp":"2024-05-17T14:52:09.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T15:48:04.048372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:10.570447+0000","last_change":"2024-05-17T14:52:10.567903+0000","last_active":"2024-05-17T14:52:10.570447+0000","last_peered":"2024-05-17T14:52:10.570447+0000","last_clean":"2024-05-17T14:52:10.570447+0000","last_became_active":"2024-05-17T14:52:10.567621+0000","last_became_peered":"2024-05-17T14:52:10.567621+0000","last_unstale":"2024-05-17T14:52:10.570447+0000","last_undegraded":"2024-05-17T14:52:10.570447+0000","last_fullsized":"2024-05-17T14:52:10.570447+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_clean_scrub_stamp":"2024-05-17T14:52:09.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-19T00:15:36.563657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:10.570401+0000","last_change":"2024-05-17T14:52:10.567390+0000","last_active":"2024-05-17T14:52:10.570401+0000","last_peered":"2024-05-17T14:52:10.570401+0000","last_clean":"2024-05-17T14:52:10.570401+0000","last_became_active":"2024-05-17T14:52:10.567087+0000","last_became_peered":"2024-05-17T14:52:10.567087+0000","last_unstale":"2024-05-17T14:52:10.570401+0000","last_undegraded":"2024-05-17T14:52:10.570401+0000","last_fullsized":"2024-05-17T14:52:10.570401+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_clean_scrub_stamp":"2024-05-17T14:52:09.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-19T01:42:59.573342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:10.568961+0000","last_change":"2024-05-17T14:52:10.564672+0000","last_active":"2024-05-17T14:52:10.568961+0000","last_peered":"2024-05-17T14:52:10.568961+0000","last_clean":"2024-05-17T14:52:10.568961+0000","last_became_active":"2024-05-17T14:52:10.564380+0000","last_became_peered":"2024-05-17T14:52:10.564380+0000","last_unstale":"2024-05-17T14:52:10.568961+0000","last_undegraded":"2024-05-17T14:52:10.568961+0000","last_fullsized":"2024-05-17T14:52:10.568961+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:09.549165+0000","last_clean_scrub_stamp":"2024-05-17T14:52:09.549165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T22:46:21.935245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":83,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:10.561959+0000","last_change":"2024-05-17T14:52:06.027203+0000","last_active":"2024-05-17T14:52:10.561959+0000","last_peered":"2024-05-17T14:52:10.561959+0000","last_clean":"2024-05-17T14:52:10.561959+0000","last_became_active":"2024-05-17T14:52:06.026327+0000","last_became_peered":"2024-05-17T14:52:06.026327+0000","last_unstale":"2024-05-17T14:52:10.561959+0000","last_undegraded":"2024-05-17T14:52:10.561959+0000","last_fullsized":"2024-05-17T14:52:10.561959+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:04.530871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:04.530871+0000","last_clean_scrub_stamp":"2024-05-17T14:52:04.530871+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T16:53:15.484878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-17T14:52:14.093 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-05-17T14:52:14.093 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-05-17T14:52:14.093 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-05-17T14:52:14.093 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-05-17T14:52:14.418 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:14.418 INFO:teuthology.orchestra.run.smithi134.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-17T14:52:14.431 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-05-17T14:52:14.431 INFO:teuthology.run_tasks:Running task ceph... 2024-05-17T14:52:14.441 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-05-17T14:52:14.441 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-17T14:52:14.441 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7711219', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing', 'sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 3754, 'sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4', 'sleep_before_teardown': 0, 'subset': '56/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4', 'targets': {'smithi134.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHYe1ZGnxZjX7+sQweRvnThF1q47PodBSwam7dcyTHgN8O9FeALEpBU/YCmwo/S2di3fktueeGSo22ojmFO2hnY=', 'smithi195.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHOwvwP+xWhFfbARMc1jjL85g4dIJywrcGaxZwuxpRdYo+R7TPpbE+bquiE8iV2Wra/zwrM6BG46opM+s86KYvw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'ee59d86aad04089481e55c1a15aa5871339935b4'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'wip-fix-extra-system-packages', 'teuthology_sha1': 'cad7edd8d899a7c59a48f7da3294fca4cb778f64', 'timestamp': '2024-05-17_14:29:27', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-17T14:52:14.442 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-05-17T14:52:14.444 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-05-17T14:52:14.462 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/run/ceph 2024-05-17T14:52:14.500 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/run/ceph 2024-05-17T14:52:14.533 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:14.533 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2024-05-17T14:52:14.592 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-17T14:52:14.592 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_1 2024-05-17T14:52:14.650 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-17T14:52:14.650 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-17T14:52:14.650 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-05-17T14:52:14.650 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-17T14:52:14.651 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-05-17T14:52:14.651 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-05-17 14:51:32.885934449 +0000 2024-05-17T14:52:14.651 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-05-17 14:48:20.565841562 +0000 2024-05-17T14:52:14.651 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-05-17 14:48:20.565841562 +0000 2024-05-17T14:52:14.651 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-05-17 14:48:20.565841562 +0000 2024-05-17T14:52:14.651 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-17T14:52:14.717 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-05-17T14:52:14.717 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-05-17T14:52:14.717 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000113325 s, 4.5 MB/s 2024-05-17T14:52:14.718 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-17T14:52:14.777 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_2 2024-05-17T14:52:14.835 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-17T14:52:14.835 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-17T14:52:14.835 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-05-17T14:52:14.835 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-17T14:52:14.835 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-05-17T14:52:14.835 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-05-17 14:51:32.885934449 +0000 2024-05-17T14:52:14.835 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-05-17 14:48:20.891835605 +0000 2024-05-17T14:52:14.836 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-05-17 14:48:20.891835605 +0000 2024-05-17T14:52:14.836 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-05-17 14:48:20.891835605 +0000 2024-05-17T14:52:14.836 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-17T14:52:14.902 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-05-17T14:52:14.902 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-05-17T14:52:14.903 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 8.4888e-05 s, 6.0 MB/s 2024-05-17T14:52:14.904 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-17T14:52:14.962 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_3 2024-05-17T14:52:15.019 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-17T14:52:15.019 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-17T14:52:15.019 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-17T14:52:15.019 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-17T14:52:15.019 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-05-17T14:52:15.019 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-05-17 14:51:32.885934449 +0000 2024-05-17T14:52:15.019 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-05-17 14:48:21.216829666 +0000 2024-05-17T14:52:15.019 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-05-17 14:48:21.216829666 +0000 2024-05-17T14:52:15.020 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-05-17 14:48:21.216829666 +0000 2024-05-17T14:52:15.020 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-17T14:52:15.084 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-05-17T14:52:15.084 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-05-17T14:52:15.084 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000119028 s, 4.3 MB/s 2024-05-17T14:52:15.085 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-17T14:52:15.142 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_4 2024-05-17T14:52:15.198 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-17T14:52:15.198 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-17T14:52:15.199 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-05-17T14:52:15.199 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-17T14:52:15.199 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-05-17T14:52:15.199 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-05-17 14:51:32.886934431 +0000 2024-05-17T14:52:15.199 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-05-17 14:48:21.603822594 +0000 2024-05-17T14:52:15.199 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-05-17 14:48:21.603822594 +0000 2024-05-17T14:52:15.199 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-05-17 14:48:21.603822594 +0000 2024-05-17T14:52:15.199 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-17T14:52:15.262 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-05-17T14:52:15.263 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-05-17T14:52:15.263 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000110253 s, 4.6 MB/s 2024-05-17T14:52:15.264 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-17T14:52:15.321 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-17T14:52:15.321 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-05-17T14:52:15.321 INFO:tasks.ceph:Generating config... 2024-05-17T14:52:15.323 INFO:tasks.ceph:[client] debug rbd = 20 2024-05-17T14:52:15.323 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-05-17T14:52:15.323 INFO:tasks.ceph:[client] log to stderr = False 2024-05-17T14:52:15.323 INFO:tasks.ceph:[client] rbd default features = 125 2024-05-17T14:52:15.323 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-05-17T14:52:15.323 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:52:15.323 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:52:15.324 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:52:15.324 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:52:15.324 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:52:15.324 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:52:15.324 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:52:15.324 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:52:15.324 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-05-17T14:52:15.324 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-05-17T14:52:15.324 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-05-17T14:52:15.324 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-17T14:52:15.324 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-17T14:52:15.324 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-17T14:52:15.324 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-17T14:52:15.324 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-17T14:52:15.324 INFO:tasks.ceph:[osd] bdev async discard = True 2024-05-17T14:52:15.325 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-05-17T14:52:15.325 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-17T14:52:15.325 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-17T14:52:15.325 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-05-17T14:52:15.325 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-05-17T14:52:15.325 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-17T14:52:15.325 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-17T14:52:15.325 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-05-17T14:52:15.325 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-17T14:52:15.325 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-17T14:52:15.325 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-17T14:52:15.325 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-17T14:52:15.325 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-05-17T14:52:15.325 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-17T14:52:15.326 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-05-17T14:52:15.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-05-17T14:52:15.401 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/cluster2.keyring 2024-05-17T14:52:15.403 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-05-17T14:52:15.486 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-05-17T14:52:15.551 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.195')] 2024-05-17T14:52:15.552 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.195', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-05-17T14:52:15.553 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:15.553 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-17T14:52:15.607 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.195 --print /home/ubuntu/cephtest/cluster2.monmap 2024-05-17T14:52:15.687 INFO:teuthology.orchestra.run.smithi195.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-05-17T14:52:15.687 INFO:teuthology.orchestra.run.smithi195.stdout:monmaptool: generated fsid 4178941f-4468-4ed6-a579-17458499f0c7 2024-05-17T14:52:15.687 INFO:teuthology.orchestra.run.smithi195.stdout:setting min_mon_release = quincy 2024-05-17T14:52:15.687 INFO:teuthology.orchestra.run.smithi195.stdout:epoch 0 2024-05-17T14:52:15.687 INFO:teuthology.orchestra.run.smithi195.stdout:fsid 4178941f-4468-4ed6-a579-17458499f0c7 2024-05-17T14:52:15.687 INFO:teuthology.orchestra.run.smithi195.stdout:last_changed 2024-05-17T14:52:15.687505+0000 2024-05-17T14:52:15.687 INFO:teuthology.orchestra.run.smithi195.stdout:created 2024-05-17T14:52:15.687505+0000 2024-05-17T14:52:15.688 INFO:teuthology.orchestra.run.smithi195.stdout:min_mon_release 17 (quincy) 2024-05-17T14:52:15.688 INFO:teuthology.orchestra.run.smithi195.stdout:election_strategy: 1 2024-05-17T14:52:15.688 INFO:teuthology.orchestra.run.smithi195.stdout:0: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.a 2024-05-17T14:52:15.688 INFO:teuthology.orchestra.run.smithi195.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-05-17T14:52:15.689 DEBUG:teuthology.orchestra.run.smithi195:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-17T14:52:15.745 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 4178941f-4468-4ed6-a579-17458499f0c7... 2024-05-17T14:52:15.746 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-05-17T14:52:15.748 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-05-17T14:52:15.796 INFO:teuthology.orchestra.run.smithi134.stdout:[global] 2024-05-17T14:52:15.797 INFO:teuthology.orchestra.run.smithi134.stdout: chdir = "" 2024-05-17T14:52:15.797 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-17T14:52:15.797 INFO:teuthology.orchestra.run.smithi134.stdout: auth supported = cephx 2024-05-17T14:52:15.797 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.797 INFO:teuthology.orchestra.run.smithi134.stdout: filestore xattr use omap = true 2024-05-17T14:52:15.797 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.797 INFO:teuthology.orchestra.run.smithi134.stdout: mon clock drift allowed = 1.000 2024-05-17T14:52:15.797 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.797 INFO:teuthology.orchestra.run.smithi134.stdout: osd crush chooseleaf type = 0 2024-05-17T14:52:15.797 INFO:teuthology.orchestra.run.smithi134.stdout: auth debug = true 2024-05-17T14:52:15.797 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.798 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on old message = true 2024-05-17T14:52:15.798 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on bug = true 2024-05-17T14:52:15.798 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.798 INFO:teuthology.orchestra.run.smithi134.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-17T14:52:15.798 INFO:teuthology.orchestra.run.smithi134.stdout: mon pg warn max object skew = 0 2024-05-17T14:52:15.798 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.798 INFO:teuthology.orchestra.run.smithi134.stdout: # disable pg_autoscaler by default for new pools 2024-05-17T14:52:15.798 INFO:teuthology.orchestra.run.smithi134.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-17T14:52:15.798 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.798 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default size = 2 2024-05-17T14:52:15.798 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.798 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow primary affinity = true 2024-05-17T14:52:15.798 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow pg remap = true 2024-05-17T14:52:15.799 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on legacy crush tunables = false 2024-05-17T14:52:15.799 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on crush straw calc version zero = false 2024-05-17T14:52:15.799 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on no sortbitwise = false 2024-05-17T14:52:15.799 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on osd down out interval zero = false 2024-05-17T14:52:15.799 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on too few osds = false 2024-05-17T14:52:15.799 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-17T14:52:15.799 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-17T14:52:15.799 INFO:teuthology.orchestra.run.smithi134.stdout: mon_allow_pool_size_one = true 2024-05-17T14:52:15.799 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.799 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-17T14:52:15.799 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.799 INFO:teuthology.orchestra.run.smithi134.stdout: osd default data pool replay window = 5 2024-05-17T14:52:15.799 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.799 INFO:teuthology.orchestra.run.smithi134.stdout: mon allow pool delete = true 2024-05-17T14:52:15.800 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.800 INFO:teuthology.orchestra.run.smithi134.stdout: mon cluster log file level = debug 2024-05-17T14:52:15.800 INFO:teuthology.orchestra.run.smithi134.stdout: debug asserts on shutdown = true 2024-05-17T14:52:15.800 INFO:teuthology.orchestra.run.smithi134.stdout: mon health detail to clog = false 2024-05-17T14:52:15.800 INFO:teuthology.orchestra.run.smithi134.stdout: mon host = 172.21.15.195 2024-05-17T14:52:15.800 INFO:teuthology.orchestra.run.smithi134.stdout: mon client directed command retry = 5 2024-05-17T14:52:15.800 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on pool no app = False 2024-05-17T14:52:15.800 INFO:teuthology.orchestra.run.smithi134.stdout: ms inject socket failures = 5000 2024-05-17T14:52:15.800 INFO:teuthology.orchestra.run.smithi134.stdout: fsid = 4178941f-4468-4ed6-a579-17458499f0c7 2024-05-17T14:52:15.800 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.800 INFO:teuthology.orchestra.run.smithi134.stdout:[osd] 2024-05-17T14:52:15.801 INFO:teuthology.orchestra.run.smithi134.stdout: osd journal size = 100 2024-05-17T14:52:15.801 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.801 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub load threshold = 5.0 2024-05-17T14:52:15.801 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub max interval = 600 2024-05-17T14:52:15.801 INFO:teuthology.orchestra.run.smithi134.stdout: osd mclock profile = high_recovery_ops 2024-05-17T14:52:15.801 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.801 INFO:teuthology.orchestra.run.smithi134.stdout: osd recover clone overlap = true 2024-05-17T14:52:15.801 INFO:teuthology.orchestra.run.smithi134.stdout: osd recovery max chunk = 1048576 2024-05-17T14:52:15.801 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.801 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug shutdown = true 2024-05-17T14:52:15.801 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug op order = true 2024-05-17T14:52:15.801 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify stray on activate = true 2024-05-17T14:52:15.801 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug trim objects = true 2024-05-17T14:52:15.801 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.802 INFO:teuthology.orchestra.run.smithi134.stdout: osd open classes on start = true 2024-05-17T14:52:15.802 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug pg log writeout = true 2024-05-17T14:52:15.802 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.802 INFO:teuthology.orchestra.run.smithi134.stdout: osd deep scrub update digest min age = 30 2024-05-17T14:52:15.802 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.802 INFO:teuthology.orchestra.run.smithi134.stdout: osd map max advance = 10 2024-05-17T14:52:15.802 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.802 INFO:teuthology.orchestra.run.smithi134.stdout: journal zero on create = true 2024-05-17T14:52:15.802 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.802 INFO:teuthology.orchestra.run.smithi134.stdout: filestore ondisk finisher threads = 3 2024-05-17T14:52:15.802 INFO:teuthology.orchestra.run.smithi134.stdout: filestore apply finisher threads = 3 2024-05-17T14:52:15.802 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.802 INFO:teuthology.orchestra.run.smithi134.stdout: bdev debug aio = true 2024-05-17T14:52:15.803 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug misdirected ops = true 2024-05-17T14:52:15.803 INFO:teuthology.orchestra.run.smithi134.stdout: bdev async discard = True 2024-05-17T14:52:15.803 INFO:teuthology.orchestra.run.smithi134.stdout: bdev enable discard = True 2024-05-17T14:52:15.803 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore block size = 96636764160 2024-05-17T14:52:15.803 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore fsck on mount = True 2024-05-17T14:52:15.803 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluefs = 1/20 2024-05-17T14:52:15.803 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluestore = 1/20 2024-05-17T14:52:15.803 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-05-17T14:52:15.803 INFO:teuthology.orchestra.run.smithi134.stdout: debug osd = 20 2024-05-17T14:52:15.803 INFO:teuthology.orchestra.run.smithi134.stdout: debug rocksdb = 4/10 2024-05-17T14:52:15.803 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd backfillfull_ratio = 0.85 2024-05-17T14:52:15.803 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd full ratio = 0.9 2024-05-17T14:52:15.803 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd nearfull ratio = 0.8 2024-05-17T14:52:15.804 INFO:teuthology.orchestra.run.smithi134.stdout: osd failsafe full ratio = 0.95 2024-05-17T14:52:15.804 INFO:teuthology.orchestra.run.smithi134.stdout: osd memory target = 1610612736 2024-05-17T14:52:15.804 INFO:teuthology.orchestra.run.smithi134.stdout: osd objectstore = bluestore 2024-05-17T14:52:15.804 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.804 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr] 2024-05-17T14:52:15.804 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-05-17T14:52:15.804 INFO:teuthology.orchestra.run.smithi134.stdout: debug mgr = 20 2024-05-17T14:52:15.804 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-05-17T14:52:15.804 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-05-17T14:52:15.804 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-05-17T14:52:15.804 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-05-17T14:52:15.804 INFO:teuthology.orchestra.run.smithi134.stdout: mgr/telemetry/nag = false 2024-05-17T14:52:15.805 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.805 INFO:teuthology.orchestra.run.smithi134.stdout:[mon] 2024-05-17T14:52:15.805 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-05-17T14:52:15.805 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-05-17T14:52:15.805 INFO:teuthology.orchestra.run.smithi134.stdout: debug paxos = 20 2024-05-17T14:52:15.805 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-05-17T14:52:15.805 INFO:teuthology.orchestra.run.smithi134.stdout: mon data avail warn = 5 2024-05-17T14:52:15.805 INFO:teuthology.orchestra.run.smithi134.stdout: mon mgr mkfs grace = 240 2024-05-17T14:52:15.805 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-05-17T14:52:15.805 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd reporter subtree level = osd 2024-05-17T14:52:15.805 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd prime pg temp = true 2024-05-17T14:52:15.805 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-05-17T14:52:15.805 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.805 INFO:teuthology.orchestra.run.smithi134.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-17T14:52:15.806 INFO:teuthology.orchestra.run.smithi134.stdout: auth mon ticket ttl = 660 # 11m 2024-05-17T14:52:15.806 INFO:teuthology.orchestra.run.smithi134.stdout: auth service ticket ttl = 240 # 4m 2024-05-17T14:52:15.806 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.806 INFO:teuthology.orchestra.run.smithi134.stdout: # don't complain about insecure global_id in the test suite 2024-05-17T14:52:15.806 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-17T14:52:15.806 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-17T14:52:15.806 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.806 INFO:teuthology.orchestra.run.smithi134.stdout: # 1m isn't quite enough 2024-05-17T14:52:15.806 INFO:teuthology.orchestra.run.smithi134.stdout: mon_down_mkfs_grace = 2m 2024-05-17T14:52:15.806 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.806 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_filestore_osds = false 2024-05-17T14:52:15.806 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-05-17T14:52:15.806 INFO:teuthology.orchestra.run.smithi134.stdout:[client] 2024-05-17T14:52:15.807 INFO:teuthology.orchestra.run.smithi134.stdout: rgw cache enabled = true 2024-05-17T14:52:15.807 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable ops log = true 2024-05-17T14:52:15.807 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable usage log = true 2024-05-17T14:52:15.807 INFO:teuthology.orchestra.run.smithi134.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-17T14:52:15.807 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-17T14:52:15.807 INFO:teuthology.orchestra.run.smithi134.stdout: debug rbd = 20 2024-05-17T14:52:15.807 INFO:teuthology.orchestra.run.smithi134.stdout: debug rbd_mirror = 15 2024-05-17T14:52:15.807 INFO:teuthology.orchestra.run.smithi134.stdout: log to stderr = False 2024-05-17T14:52:15.807 INFO:teuthology.orchestra.run.smithi134.stdout: rbd default features = 125 2024-05-17T14:52:15.807 INFO:teuthology.orchestra.run.smithi134.stdout: rbd mirror image policy type = simple 2024-05-17T14:52:15.807 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.a] 2024-05-17T14:52:15.807 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.0] 2024-05-17T14:52:15.807 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:52:15.807 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:52:15.807 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.1] 2024-05-17T14:52:15.808 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:52:15.808 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:52:15.808 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.2] 2024-05-17T14:52:15.808 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:52:15.808 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:52:15.808 INFO:teuthology.orchestra.run.smithi134.stdout:[client.mirror.3] 2024-05-17T14:52:15.808 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:52:15.808 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:52:15.828 INFO:teuthology.orchestra.run.smithi195.stdout:[global] 2024-05-17T14:52:15.828 INFO:teuthology.orchestra.run.smithi195.stdout: chdir = "" 2024-05-17T14:52:15.829 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-17T14:52:15.829 INFO:teuthology.orchestra.run.smithi195.stdout: auth supported = cephx 2024-05-17T14:52:15.829 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.829 INFO:teuthology.orchestra.run.smithi195.stdout: filestore xattr use omap = true 2024-05-17T14:52:15.829 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.829 INFO:teuthology.orchestra.run.smithi195.stdout: mon clock drift allowed = 1.000 2024-05-17T14:52:15.829 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.829 INFO:teuthology.orchestra.run.smithi195.stdout: osd crush chooseleaf type = 0 2024-05-17T14:52:15.829 INFO:teuthology.orchestra.run.smithi195.stdout: auth debug = true 2024-05-17T14:52:15.829 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.829 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on old message = true 2024-05-17T14:52:15.829 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on bug = true 2024-05-17T14:52:15.829 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.830 INFO:teuthology.orchestra.run.smithi195.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-17T14:52:15.830 INFO:teuthology.orchestra.run.smithi195.stdout: mon pg warn max object skew = 0 2024-05-17T14:52:15.830 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.830 INFO:teuthology.orchestra.run.smithi195.stdout: # disable pg_autoscaler by default for new pools 2024-05-17T14:52:15.830 INFO:teuthology.orchestra.run.smithi195.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-17T14:52:15.830 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.830 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default size = 2 2024-05-17T14:52:15.830 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.830 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow primary affinity = true 2024-05-17T14:52:15.830 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow pg remap = true 2024-05-17T14:52:15.830 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on legacy crush tunables = false 2024-05-17T14:52:15.830 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on crush straw calc version zero = false 2024-05-17T14:52:15.830 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on no sortbitwise = false 2024-05-17T14:52:15.831 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on osd down out interval zero = false 2024-05-17T14:52:15.831 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on too few osds = false 2024-05-17T14:52:15.831 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-17T14:52:15.831 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-17T14:52:15.831 INFO:teuthology.orchestra.run.smithi195.stdout: mon_allow_pool_size_one = true 2024-05-17T14:52:15.831 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.831 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-17T14:52:15.831 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.831 INFO:teuthology.orchestra.run.smithi195.stdout: osd default data pool replay window = 5 2024-05-17T14:52:15.831 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.831 INFO:teuthology.orchestra.run.smithi195.stdout: mon allow pool delete = true 2024-05-17T14:52:15.831 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.831 INFO:teuthology.orchestra.run.smithi195.stdout: mon cluster log file level = debug 2024-05-17T14:52:15.831 INFO:teuthology.orchestra.run.smithi195.stdout: debug asserts on shutdown = true 2024-05-17T14:52:15.832 INFO:teuthology.orchestra.run.smithi195.stdout: mon health detail to clog = false 2024-05-17T14:52:15.832 INFO:teuthology.orchestra.run.smithi195.stdout: mon host = 172.21.15.195 2024-05-17T14:52:15.832 INFO:teuthology.orchestra.run.smithi195.stdout: mon client directed command retry = 5 2024-05-17T14:52:15.832 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on pool no app = False 2024-05-17T14:52:15.832 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject socket failures = 5000 2024-05-17T14:52:15.832 INFO:teuthology.orchestra.run.smithi195.stdout: fsid = 4178941f-4468-4ed6-a579-17458499f0c7 2024-05-17T14:52:15.832 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.832 INFO:teuthology.orchestra.run.smithi195.stdout:[osd] 2024-05-17T14:52:15.832 INFO:teuthology.orchestra.run.smithi195.stdout: osd journal size = 100 2024-05-17T14:52:15.832 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.832 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub load threshold = 5.0 2024-05-17T14:52:15.833 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub max interval = 600 2024-05-17T14:52:15.833 INFO:teuthology.orchestra.run.smithi195.stdout: osd mclock profile = high_recovery_ops 2024-05-17T14:52:15.833 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.833 INFO:teuthology.orchestra.run.smithi195.stdout: osd recover clone overlap = true 2024-05-17T14:52:15.833 INFO:teuthology.orchestra.run.smithi195.stdout: osd recovery max chunk = 1048576 2024-05-17T14:52:15.833 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.833 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug shutdown = true 2024-05-17T14:52:15.833 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug op order = true 2024-05-17T14:52:15.834 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify stray on activate = true 2024-05-17T14:52:15.834 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug trim objects = true 2024-05-17T14:52:15.834 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.834 INFO:teuthology.orchestra.run.smithi195.stdout: osd open classes on start = true 2024-05-17T14:52:15.834 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug pg log writeout = true 2024-05-17T14:52:15.834 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.834 INFO:teuthology.orchestra.run.smithi195.stdout: osd deep scrub update digest min age = 30 2024-05-17T14:52:15.834 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.834 INFO:teuthology.orchestra.run.smithi195.stdout: osd map max advance = 10 2024-05-17T14:52:15.834 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.835 INFO:teuthology.orchestra.run.smithi195.stdout: journal zero on create = true 2024-05-17T14:52:15.835 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.835 INFO:teuthology.orchestra.run.smithi195.stdout: filestore ondisk finisher threads = 3 2024-05-17T14:52:15.835 INFO:teuthology.orchestra.run.smithi195.stdout: filestore apply finisher threads = 3 2024-05-17T14:52:15.835 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.835 INFO:teuthology.orchestra.run.smithi195.stdout: bdev debug aio = true 2024-05-17T14:52:15.835 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug misdirected ops = true 2024-05-17T14:52:15.835 INFO:teuthology.orchestra.run.smithi195.stdout: bdev async discard = True 2024-05-17T14:52:15.835 INFO:teuthology.orchestra.run.smithi195.stdout: bdev enable discard = True 2024-05-17T14:52:15.835 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore block size = 96636764160 2024-05-17T14:52:15.835 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore fsck on mount = True 2024-05-17T14:52:15.835 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluefs = 1/20 2024-05-17T14:52:15.836 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluestore = 1/20 2024-05-17T14:52:15.836 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-05-17T14:52:15.836 INFO:teuthology.orchestra.run.smithi195.stdout: debug osd = 20 2024-05-17T14:52:15.836 INFO:teuthology.orchestra.run.smithi195.stdout: debug rocksdb = 4/10 2024-05-17T14:52:15.836 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd backfillfull_ratio = 0.85 2024-05-17T14:52:15.836 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd full ratio = 0.9 2024-05-17T14:52:15.836 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd nearfull ratio = 0.8 2024-05-17T14:52:15.836 INFO:teuthology.orchestra.run.smithi195.stdout: osd failsafe full ratio = 0.95 2024-05-17T14:52:15.836 INFO:teuthology.orchestra.run.smithi195.stdout: osd memory target = 1610612736 2024-05-17T14:52:15.836 INFO:teuthology.orchestra.run.smithi195.stdout: osd objectstore = bluestore 2024-05-17T14:52:15.836 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.836 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr] 2024-05-17T14:52:15.836 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-05-17T14:52:15.836 INFO:teuthology.orchestra.run.smithi195.stdout: debug mgr = 20 2024-05-17T14:52:15.837 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2024-05-17T14:52:15.837 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2024-05-17T14:52:15.837 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2024-05-17T14:52:15.837 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2024-05-17T14:52:15.837 INFO:teuthology.orchestra.run.smithi195.stdout: mgr/telemetry/nag = false 2024-05-17T14:52:15.837 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.837 INFO:teuthology.orchestra.run.smithi195.stdout:[mon] 2024-05-17T14:52:15.837 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-05-17T14:52:15.837 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2024-05-17T14:52:15.837 INFO:teuthology.orchestra.run.smithi195.stdout: debug paxos = 20 2024-05-17T14:52:15.837 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2024-05-17T14:52:15.837 INFO:teuthology.orchestra.run.smithi195.stdout: mon data avail warn = 5 2024-05-17T14:52:15.837 INFO:teuthology.orchestra.run.smithi195.stdout: mon mgr mkfs grace = 240 2024-05-17T14:52:15.837 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2024-05-17T14:52:15.838 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd reporter subtree level = osd 2024-05-17T14:52:15.838 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd prime pg temp = true 2024-05-17T14:52:15.838 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2024-05-17T14:52:15.838 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.838 INFO:teuthology.orchestra.run.smithi195.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-17T14:52:15.838 INFO:teuthology.orchestra.run.smithi195.stdout: auth mon ticket ttl = 660 # 11m 2024-05-17T14:52:15.838 INFO:teuthology.orchestra.run.smithi195.stdout: auth service ticket ttl = 240 # 4m 2024-05-17T14:52:15.839 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.839 INFO:teuthology.orchestra.run.smithi195.stdout: # don't complain about insecure global_id in the test suite 2024-05-17T14:52:15.839 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-17T14:52:15.839 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-17T14:52:15.839 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.839 INFO:teuthology.orchestra.run.smithi195.stdout: # 1m isn't quite enough 2024-05-17T14:52:15.839 INFO:teuthology.orchestra.run.smithi195.stdout: mon_down_mkfs_grace = 2m 2024-05-17T14:52:15.839 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.839 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_filestore_osds = false 2024-05-17T14:52:15.839 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:15.839 INFO:teuthology.orchestra.run.smithi195.stdout:[client] 2024-05-17T14:52:15.840 INFO:teuthology.orchestra.run.smithi195.stdout: rgw cache enabled = true 2024-05-17T14:52:15.840 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable ops log = true 2024-05-17T14:52:15.840 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable usage log = true 2024-05-17T14:52:15.840 INFO:teuthology.orchestra.run.smithi195.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-17T14:52:15.840 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-17T14:52:15.840 INFO:teuthology.orchestra.run.smithi195.stdout: debug rbd = 20 2024-05-17T14:52:15.840 INFO:teuthology.orchestra.run.smithi195.stdout: debug rbd_mirror = 15 2024-05-17T14:52:15.840 INFO:teuthology.orchestra.run.smithi195.stdout: log to stderr = False 2024-05-17T14:52:15.840 INFO:teuthology.orchestra.run.smithi195.stdout: rbd default features = 125 2024-05-17T14:52:15.840 INFO:teuthology.orchestra.run.smithi195.stdout: rbd mirror image policy type = simple 2024-05-17T14:52:15.840 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.a] 2024-05-17T14:52:15.840 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.0] 2024-05-17T14:52:15.840 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:52:15.840 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:52:15.841 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.1] 2024-05-17T14:52:15.841 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:52:15.841 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:52:15.841 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.2] 2024-05-17T14:52:15.841 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:52:15.841 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:52:15.841 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.3] 2024-05-17T14:52:15.841 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-05-17T14:52:15.841 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-05-17T14:52:15.842 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-05-17T14:52:15.842 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-05-17T14:52:15.936 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-17T14:52:15.936 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:15.937 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-05-17T14:52:15.952 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:15.952 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-05-17T14:52:16.008 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi134.front.sepia.ceph.com 2024-05-17T14:52:16.008 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:52:16.008 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/cluster2.keyring 2024-05-17T14:52:16.008 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-05-17T14:52:16.048 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:52:16.048 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-05-17T14:52:16.106 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi195.front.sepia.ceph.com 2024-05-17T14:52:16.106 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:16.106 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/cluster2.keyring 2024-05-17T14:52:16.106 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-05-17T14:52:16.140 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:16.141 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-05-17T14:52:16.196 INFO:tasks.ceph:Setting up mon nodes... 2024-05-17T14:52:16.196 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-17T14:52:16.196 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-05-17T14:52:16.252 INFO:teuthology.orchestra.run.smithi134.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-05-17T14:52:16.254 INFO:tasks.ceph:Setting up mds nodes... 2024-05-17T14:52:16.254 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-17T14:52:16.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-05-17T14:52:16.296 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-05-17T14:52:16.309 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-05-17T14:52:16.349 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-05-17T14:52:16.362 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-05-17T14:52:16.443 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-05-17T14:52:16.455 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-05-17T14:52:16.536 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-05-17T14:52:16.548 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-05-17T14:52:16.629 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-05-17T14:52:16.641 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-05-17T14:52:16.722 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-05-17T14:52:16.734 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-05-17T14:52:16.815 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-05-17T14:52:16.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-05-17T14:52:16.909 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-05-17T14:52:16.922 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-05-17T14:52:17.003 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-05-17T14:52:17.016 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-17T14:52:17.016 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-05-17T14:52:17.016 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-05-17T14:52:17.082 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-17T14:52:17.082 INFO:tasks.ceph:role: cluster2.osd.0 2024-05-17T14:52:17.082 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com 2024-05-17T14:52:17.082 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-17T14:52:17.146 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-17T14:52:17.146 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-17T14:52:17.146 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-17T14:52:17.147 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-17T14:52:17.147 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-17T14:52:17.147 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-05-17T14:52:17.147 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-17T14:52:17.147 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-17T14:52:17.147 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-17T14:52:17.147 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-17T14:52:17.181 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-05-17T14:52:17.183 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-05-17T14:52:17.184 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-05-17T14:52:17.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-05-17T14:52:17.321 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-05-17T14:52:17.387 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-17T14:52:17.387 INFO:tasks.ceph:role: cluster2.osd.1 2024-05-17T14:52:17.387 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com 2024-05-17T14:52:17.387 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-17T14:52:17.451 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-17T14:52:17.451 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-17T14:52:17.451 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-17T14:52:17.451 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-17T14:52:17.452 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-17T14:52:17.452 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-05-17T14:52:17.452 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-17T14:52:17.452 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-17T14:52:17.452 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-17T14:52:17.452 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-17T14:52:17.487 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-05-17T14:52:17.489 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-05-17T14:52:17.489 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-05-17T14:52:17.565 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-05-17T14:52:17.635 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-05-17T14:52:17.701 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-05-17T14:52:17.701 INFO:tasks.ceph:role: cluster2.osd.2 2024-05-17T14:52:17.701 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com 2024-05-17T14:52:17.701 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-17T14:52:17.766 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-17T14:52:17.766 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-17T14:52:17.766 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-17T14:52:17.766 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-17T14:52:17.766 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-17T14:52:17.766 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-05-17T14:52:17.766 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-17T14:52:17.766 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-17T14:52:17.767 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-17T14:52:17.767 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-17T14:52:17.801 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-05-17T14:52:17.803 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-05-17T14:52:17.803 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-05-17T14:52:17.878 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-05-17T14:52:17.948 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-05-17T14:52:18.050 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:18.049+0000 7f0f60e1b640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-05-17T14:52:18.050 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:18.049+0000 7f0f60e1b640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-05-17T14:52:18.050 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:18.049+0000 7f0f60e1b640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-05-17T14:52:18.050 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:18.049+0000 7f0f60e1b640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-05-17T14:52:18.050 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:18.049+0000 7f0f60e1b640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-05-17T14:52:18.050 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:18.049+0000 7f0f60e1b640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-05-17T14:52:20.906 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-05-17T14:52:20.936 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-05-17T14:52:21.027 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:21.026+0000 7f52ea2aa640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-05-17T14:52:21.027 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:21.026+0000 7f52ea2aa640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-05-17T14:52:21.027 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:21.026+0000 7f52ea2aa640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-05-17T14:52:21.027 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:21.026+0000 7f52ea2aa640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-05-17T14:52:21.027 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:21.026+0000 7f52ea2aa640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-05-17T14:52:21.027 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:21.026+0000 7f52ea2aa640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-05-17T14:52:23.879 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-05-17T14:52:23.906 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-05-17T14:52:23.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:23.994+0000 7f7ebf098640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-05-17T14:52:23.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:23.994+0000 7f7ebf098640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-05-17T14:52:23.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:23.994+0000 7f7ebf098640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-05-17T14:52:23.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:23.994+0000 7f7ebf098640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-05-17T14:52:23.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:23.994+0000 7f7ebf098640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-05-17T14:52:23.996 INFO:teuthology.orchestra.run.smithi195.stderr:2024-05-17T14:52:23.994+0000 7f7ebf098640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-05-17T14:52:26.838 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-05-17T14:52:26.868 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-17T14:52:26.868 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:52:26.868 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-05-17T14:52:26.899 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:26.899 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-05-17T14:52:26.936 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:26.936 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-05-17T14:52:27.003 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:27.003 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-05-17T14:52:27.070 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-05-17T14:52:27.070 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-05-17T14:52:27.089 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:27.089 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-05-17T14:52:27.126 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:27.127 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-05-17T14:52:27.182 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:27.182 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-05-17T14:52:27.238 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:27.238 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-05-17T14:52:27.294 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:27.294 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-05-17T14:52:27.350 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:27.350 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-05-17T14:52:27.406 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:27.406 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-05-17T14:52:27.461 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:27.461 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-05-17T14:52:27.516 INFO:tasks.ceph:Adding keys to all mons... 2024-05-17T14:52:27.516 DEBUG:teuthology.orchestra.run.smithi195:> sudo tee -a /etc/ceph/cluster2.keyring 2024-05-17T14:52:27.579 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr.x] 2024-05-17T14:52:27.579 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAgb0dmMUAJDxAAkrv0my3qJdTwiR/e1RujkQ== 2024-05-17T14:52:27.579 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.0] 2024-05-17T14:52:27.579 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAib0dmB5j9AhAApBF0XBROHQtPP3Z5unjAyA== 2024-05-17T14:52:27.579 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.1] 2024-05-17T14:52:27.579 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAlb0dmIR+bARAA4DuFpihPM2lyZXdHhfb6NA== 2024-05-17T14:52:27.580 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.2] 2024-05-17T14:52:27.580 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAnb0dmMv5gOxAAO6x8bJf4/NVYVdnOWsARoQ== 2024-05-17T14:52:27.580 INFO:teuthology.orchestra.run.smithi195.stdout:[client.0] 2024-05-17T14:52:27.580 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAgb0dmFu2xERAAwIypyYT2m7Zvxjcl3qrFIA== 2024-05-17T14:52:27.580 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror] 2024-05-17T14:52:27.580 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAgb0dmUszgFBAATPIA95op0RTQIojqzNOpVQ== 2024-05-17T14:52:27.580 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.0] 2024-05-17T14:52:27.580 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAgb0dmuXRwGhAAglf+QfVGJBKqwJPfuP1zag== 2024-05-17T14:52:27.580 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.1] 2024-05-17T14:52:27.580 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAgb0dmzWL8HxAA3V2YAxOSIWWCHJwn6O1u6w== 2024-05-17T14:52:27.580 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.2] 2024-05-17T14:52:27.580 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAgb0dmwFiDJRAAxzNXgUmkbVDVUZY0YJZJNA== 2024-05-17T14:52:27.580 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.3] 2024-05-17T14:52:27.580 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAgb0dmOIUQKxAAew7c+I8XAd4zoehjsLMl2w== 2024-05-17T14:52:27.581 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.4] 2024-05-17T14:52:27.581 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAgb0dmYAmmMBAAuYSPrXgIZV7oaq+GLIOA5w== 2024-05-17T14:52:27.581 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.5] 2024-05-17T14:52:27.581 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAgb0dmqts+NhAApLcKaXCshDa0kBoyMZyJOQ== 2024-05-17T14:52:27.581 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.6] 2024-05-17T14:52:27.581 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAhb0dmLEI+ABAA2gavP5ncVmyWoURPdjTv0A== 2024-05-17T14:52:27.582 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-17T14:52:27.673 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-17T14:52:27.724 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-17T14:52:27.774 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-17T14:52:27.824 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:52:27.877 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:52:27.931 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:52:27.984 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:52:28.037 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:52:28.091 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:52:28.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:52:28.198 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:52:28.252 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-17T14:52:28.307 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-17T14:52:28.307 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-05-17T14:52:28.332 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-05-17T14:52:28.427 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-05-17T14:52:28.451 DEBUG:teuthology.orchestra.run.smithi195:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-05-17T14:52:28.506 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-05-17T14:52:28.507 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-05-17T14:52:28.507 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-05-17T14:52:28.549 INFO:tasks.ceph.cluster2.mon.a:Started 2024-05-17T14:52:28.549 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-05-17T14:52:28.549 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-05-17T14:52:28.549 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-05-17T14:52:28.552 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-05-17T14:52:28.552 INFO:tasks.ceph:Setting crush tunables to default 2024-05-17T14:52:28.552 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster cluster2 osd crush tunables default 2024-05-17T14:52:28.716 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:28.715+0000 7fde5d838040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-17T14:52:28.723 INFO:teuthology.orchestra.run.smithi195.stderr:adjusted tunables profile to default 2024-05-17T14:52:28.734 INFO:tasks.ceph:check_enable_crimson: False 2024-05-17T14:52:28.735 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-05-17T14:52:28.735 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:28.735 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-05-17T14:52:28.762 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:28.763 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-05-17T14:52:28.828 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:28.828+0000 7fde5d838040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-17T14:52:28.829 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:52:28.829 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-05-17T14:52:28.875 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:28.875+0000 7fde5d838040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-17T14:52:28.894 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster cluster2 osd new 6992ef93-23af-4c62-a87c-69928173129d 0 2024-05-17T14:52:29.078 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:29.077+0000 7fde5d838040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-17T14:52:29.109 INFO:teuthology.orchestra.run.smithi195.stdout:0 2024-05-17T14:52:29.120 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster cluster2 osd new 7cab2a1d-5830-4f52-9e22-820252741227 1 2024-05-17T14:52:29.121 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:29.121+0000 7fde5d838040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-17T14:52:29.295 INFO:teuthology.orchestra.run.smithi195.stdout:1 2024-05-17T14:52:29.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster cluster2 osd new d58fe712-6edb-4306-8ff0-df9a8fc43088 2 2024-05-17T14:52:29.483 INFO:teuthology.orchestra.run.smithi195.stdout:2 2024-05-17T14:52:29.495 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-05-17T14:52:29.495 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-05-17T14:52:29.497 INFO:tasks.ceph.cluster2.osd.0:Started 2024-05-17T14:52:29.497 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-05-17T14:52:29.497 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-05-17T14:52:29.499 INFO:tasks.ceph.cluster2.osd.1:Started 2024-05-17T14:52:29.499 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-05-17T14:52:29.500 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-05-17T14:52:29.501 INFO:tasks.ceph.cluster2.osd.2:Started 2024-05-17T14:52:29.502 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-05-17T14:52:29.601 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:29.601+0000 7fde5d838040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-17T14:52:29.675 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:29.675 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":5,"fsid":"4178941f-4468-4ed6-a579-17458499f0c7","created":"2024-05-17T14:52:28.634707+0000","modified":"2024-05-17T14:52:29.483174+0000","last_up_change":"0.000000","last_in_change":"2024-05-17T14:52:29.483174+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6992ef93-23af-4c62-a87c-69928173129d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7cab2a1d-5830-4f52-9e22-820252741227","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d58fe712-6edb-4306-8ff0-df9a8fc43088","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-17T14:52:29.685 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:29.684+0000 7fde5d838040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-17T14:52:29.686 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-05-17T14:52:29.686 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-17T14:52:29.726 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:29.725+0000 7fde5d838040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-17T14:52:29.887 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:29.886+0000 7fde5d838040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-17T14:52:29.931 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:29.930+0000 7fde5d838040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-17T14:52:29.979 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:29.979+0000 7fde5d838040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-17T14:52:29.990 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-05-17T14:52:30.089 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:30.088+0000 7fde5d838040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-17T14:52:30.091 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:2024-05-17T14:52:30.090+0000 7f939450f640 -1 Falling back to public interface 2024-05-17T14:52:30.091 INFO:tasks.ceph.cluster2.osd.1.smithi195.stderr:2024-05-17T14:52:30.090+0000 7f83ace26640 -1 Falling back to public interface 2024-05-17T14:52:30.092 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:2024-05-17T14:52:30.090+0000 7f93934be640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2024-05-17T14:52:30.092 INFO:tasks.ceph.cluster2.osd.1.smithi195.stderr:2024-05-17T14:52:30.090+0000 7f83ab82e640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2024-05-17T14:52:30.092 INFO:tasks.ceph.cluster2.osd.1.smithi195.stderr:2024-05-17T14:52:30.090+0000 7f83ab82e640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6802/0: (98) Address already in use 2024-05-17T14:52:30.094 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T14:52:30.093+0000 7f7a5e20c640 -1 Falling back to public interface 2024-05-17T14:52:30.151 INFO:teuthology.misc.health.smithi195.stdout: 2024-05-17T14:52:30.151 INFO:teuthology.misc.health.smithi195.stdout:{"epoch":5,"fsid":"4178941f-4468-4ed6-a579-17458499f0c7","created":"2024-05-17T14:52:28.634707+0000","modified":"2024-05-17T14:52:29.483174+0000","last_up_change":"0.000000","last_in_change":"2024-05-17T14:52:29.483174+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6992ef93-23af-4c62-a87c-69928173129d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7cab2a1d-5830-4f52-9e22-820252741227","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d58fe712-6edb-4306-8ff0-df9a8fc43088","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-17T14:52:30.161 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-05-17T14:52:30.405 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:30.404+0000 7fde5d838040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-17T14:52:30.527 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:30.526+0000 7fde5d838040 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-17T14:52:30.737 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:30.736+0000 7fde5d838040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-17T14:52:30.786 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:30.786+0000 7fde5d838040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-17T14:52:30.899 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:30.898+0000 7fde5d838040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-17T14:52:31.164 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:31.163+0000 7fde5d838040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-17T14:52:31.208 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:31.207+0000 7fde5d838040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-17T14:52:31.351 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:31.350+0000 7fde5d838040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-17T14:52:31.484 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:31.483+0000 7fde5d838040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-17T14:52:31.532 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:31.531+0000 7fde5d838040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-17T14:52:31.586 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:31.585+0000 7fde5d838040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-17T14:52:31.629 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:31.629+0000 7fde5d838040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-17T14:52:32.022 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:32.022+0000 7fde5d838040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-17T14:52:32.137 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:32.137+0000 7fde5d838040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-17T14:52:33.559 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:2024-05-17T14:52:33.558+0000 7f939450f640 -1 osd.0 0 log_to_monitors true 2024-05-17T14:52:33.601 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T14:52:33.600+0000 7f7a5e20c640 -1 osd.2 0 log_to_monitors true 2024-05-17T14:52:33.617 INFO:tasks.ceph.cluster2.osd.1.smithi195.stderr:2024-05-17T14:52:33.617+0000 7f83ace26640 -1 osd.1 0 log_to_monitors true 2024-05-17T14:52:35.148 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T14:52:35.147+0000 7fddfd398640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-05-17T14:52:35.665 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T14:52:35.665+0000 7f7a5a1a2640 -1 osd.2 0 waiting for initial osdmap 2024-05-17T14:52:35.682 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T14:52:35.682+0000 7f7a51fc2640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-17T14:52:35.683 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:2024-05-17T14:52:35.682+0000 7f9390cb9640 -1 osd.0 0 waiting for initial osdmap 2024-05-17T14:52:35.700 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:2024-05-17T14:52:35.700+0000 7f93882c5640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-17T14:52:36.439 INFO:tasks.ceph.cluster2.osd.1.smithi195.stderr:2024-05-17T14:52:36.439+0000 7f83a8815640 -1 osd.1 0 waiting for initial osdmap 2024-05-17T14:52:36.453 INFO:tasks.ceph.cluster2.osd.1.smithi195.stderr:2024-05-17T14:52:36.453+0000 7f83a0635640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-17T14:52:36.465 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-05-17T14:52:36.754 INFO:teuthology.misc.health.smithi195.stdout: 2024-05-17T14:52:36.754 INFO:teuthology.misc.health.smithi195.stdout:{"epoch":9,"fsid":"4178941f-4468-4ed6-a579-17458499f0c7","created":"2024-05-17T14:52:28.634707+0000","modified":"2024-05-17T14:52:36.666893+0000","last_up_change":"2024-05-17T14:52:36.666893+0000","last_in_change":"2024-05-17T14:52:29.483174+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6992ef93-23af-4c62-a87c-69928173129d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6801","nonce":3281477794},{"type":"v1","addr":"172.21.15.195:6802","nonce":3281477794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3281477794},{"type":"v1","addr":"172.21.15.195:6806","nonce":3281477794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":3281477794},{"type":"v1","addr":"172.21.15.195:6812","nonce":3281477794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6807","nonce":3281477794},{"type":"v1","addr":"172.21.15.195:6809","nonce":3281477794}]},"public_addr":"172.21.15.195:6802/3281477794","cluster_addr":"172.21.15.195:6806/3281477794","heartbeat_back_addr":"172.21.15.195:6812/3281477794","heartbeat_front_addr":"172.21.15.195:6809/3281477794","state":["exists","up"]},{"osd":1,"uuid":"7cab2a1d-5830-4f52-9e22-820252741227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":1834178186},{"type":"v1","addr":"172.21.15.195:6803","nonce":1834178186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6805","nonce":1834178186},{"type":"v1","addr":"172.21.15.195:6808","nonce":1834178186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1834178186},{"type":"v1","addr":"172.21.15.195:6815","nonce":1834178186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1834178186},{"type":"v1","addr":"172.21.15.195:6813","nonce":1834178186}]},"public_addr":"172.21.15.195:6803/1834178186","cluster_addr":"172.21.15.195:6808/1834178186","heartbeat_back_addr":"172.21.15.195:6815/1834178186","heartbeat_front_addr":"172.21.15.195:6813/1834178186","state":["exists","up"]},{"osd":2,"uuid":"d58fe712-6edb-4306-8ff0-df9a8fc43088","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":821328372},{"type":"v1","addr":"172.21.15.195:6817","nonce":821328372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":821328372},{"type":"v1","addr":"172.21.15.195:6819","nonce":821328372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":821328372},{"type":"v1","addr":"172.21.15.195:6823","nonce":821328372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":821328372},{"type":"v1","addr":"172.21.15.195:6821","nonce":821328372}]},"public_addr":"172.21.15.195:6817/821328372","cluster_addr":"172.21.15.195:6819/821328372","heartbeat_back_addr":"172.21.15.195:6823/821328372","heartbeat_front_addr":"172.21.15.195:6821/821328372","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-17T14:52:36.766 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-05-17T14:52:36.766 INFO:tasks.ceph:Creating RBD pool 2024-05-17T14:52:36.766 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-05-17T14:52:37.686 INFO:teuthology.orchestra.run.smithi195.stderr:pool 'rbd' created 2024-05-17T14:52:37.848 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-05-17T14:52:38.688 INFO:teuthology.orchestra.run.smithi195.stderr:enabled application 'rbd' on pool 'rbd' 2024-05-17T14:52:38.707 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-05-17T14:52:38.708 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-05-17T14:52:38.708 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-05-17T14:52:38.708 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-05-17T14:52:38.709 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-17T14:52:39.049 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:39.061 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":4,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":3925353641},{"type":"v1","addr":"172.21.15.134:6827","nonce":3925353641}]},"active_addr":"172.21.15.134:6827/3925353641","active_change":"2024-05-17T14:52:34.139873+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":3289286487}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":1416154515}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":3241706273}]}]} 2024-05-17T14:52:39.064 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-05-17T14:52:39.064 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-05-17T14:52:39.064 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-05-17T14:52:39.356 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:39.356 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":11,"fsid":"4178941f-4468-4ed6-a579-17458499f0c7","created":"2024-05-17T14:52:28.634707+0000","modified":"2024-05-17T14:52:38.686503+0000","last_up_change":"2024-05-17T14:52:36.666893+0000","last_in_change":"2024-05-17T14:52:29.483174+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-05-17T14:52:37.062277+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-05-17T14:52:38.161835+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ef93-23af-4c62-a87c-69928173129d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6801","nonce":3281477794},{"type":"v1","addr":"172.21.15.195:6802","nonce":3281477794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3281477794},{"type":"v1","addr":"172.21.15.195:6806","nonce":3281477794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":3281477794},{"type":"v1","addr":"172.21.15.195:6812","nonce":3281477794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6807","nonce":3281477794},{"type":"v1","addr":"172.21.15.195:6809","nonce":3281477794}]},"public_addr":"172.21.15.195:6802/3281477794","cluster_addr":"172.21.15.195:6806/3281477794","heartbeat_back_addr":"172.21.15.195:6812/3281477794","heartbeat_front_addr":"172.21.15.195:6809/3281477794","state":["exists","up"]},{"osd":1,"uuid":"7cab2a1d-5830-4f52-9e22-820252741227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":1834178186},{"type":"v1","addr":"172.21.15.195:6803","nonce":1834178186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6805","nonce":1834178186},{"type":"v1","addr":"172.21.15.195:6808","nonce":1834178186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1834178186},{"type":"v1","addr":"172.21.15.195:6815","nonce":1834178186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1834178186},{"type":"v1","addr":"172.21.15.195:6813","nonce":1834178186}]},"public_addr":"172.21.15.195:6803/1834178186","cluster_addr":"172.21.15.195:6808/1834178186","heartbeat_back_addr":"172.21.15.195:6815/1834178186","heartbeat_front_addr":"172.21.15.195:6813/1834178186","state":["exists","up"]},{"osd":2,"uuid":"d58fe712-6edb-4306-8ff0-df9a8fc43088","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":821328372},{"type":"v1","addr":"172.21.15.195:6817","nonce":821328372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":821328372},{"type":"v1","addr":"172.21.15.195:6819","nonce":821328372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":821328372},{"type":"v1","addr":"172.21.15.195:6823","nonce":821328372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":821328372},{"type":"v1","addr":"172.21.15.195:6821","nonce":821328372}]},"public_addr":"172.21.15.195:6817/821328372","cluster_addr":"172.21.15.195:6819/821328372","heartbeat_back_addr":"172.21.15.195:6823/821328372","heartbeat_front_addr":"172.21.15.195:6821/821328372","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-17T14:52:34.596386+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-17T14:52:34.584549+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-17T14:52:35.666237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-17T14:52:39.369 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-05-17T14:52:39.369 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-05-17T14:52:39.660 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:39.660 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":11,"fsid":"4178941f-4468-4ed6-a579-17458499f0c7","created":"2024-05-17T14:52:28.634707+0000","modified":"2024-05-17T14:52:38.686503+0000","last_up_change":"2024-05-17T14:52:36.666893+0000","last_in_change":"2024-05-17T14:52:29.483174+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-05-17T14:52:37.062277+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-05-17T14:52:38.161835+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ef93-23af-4c62-a87c-69928173129d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6801","nonce":3281477794},{"type":"v1","addr":"172.21.15.195:6802","nonce":3281477794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3281477794},{"type":"v1","addr":"172.21.15.195:6806","nonce":3281477794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6811","nonce":3281477794},{"type":"v1","addr":"172.21.15.195:6812","nonce":3281477794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6807","nonce":3281477794},{"type":"v1","addr":"172.21.15.195:6809","nonce":3281477794}]},"public_addr":"172.21.15.195:6802/3281477794","cluster_addr":"172.21.15.195:6806/3281477794","heartbeat_back_addr":"172.21.15.195:6812/3281477794","heartbeat_front_addr":"172.21.15.195:6809/3281477794","state":["exists","up"]},{"osd":1,"uuid":"7cab2a1d-5830-4f52-9e22-820252741227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":1834178186},{"type":"v1","addr":"172.21.15.195:6803","nonce":1834178186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6805","nonce":1834178186},{"type":"v1","addr":"172.21.15.195:6808","nonce":1834178186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":1834178186},{"type":"v1","addr":"172.21.15.195:6815","nonce":1834178186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":1834178186},{"type":"v1","addr":"172.21.15.195:6813","nonce":1834178186}]},"public_addr":"172.21.15.195:6803/1834178186","cluster_addr":"172.21.15.195:6808/1834178186","heartbeat_back_addr":"172.21.15.195:6815/1834178186","heartbeat_front_addr":"172.21.15.195:6813/1834178186","state":["exists","up"]},{"osd":2,"uuid":"d58fe712-6edb-4306-8ff0-df9a8fc43088","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":821328372},{"type":"v1","addr":"172.21.15.195:6817","nonce":821328372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":821328372},{"type":"v1","addr":"172.21.15.195:6819","nonce":821328372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":821328372},{"type":"v1","addr":"172.21.15.195:6823","nonce":821328372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":821328372},{"type":"v1","addr":"172.21.15.195:6821","nonce":821328372}]},"public_addr":"172.21.15.195:6817/821328372","cluster_addr":"172.21.15.195:6819/821328372","heartbeat_back_addr":"172.21.15.195:6823/821328372","heartbeat_front_addr":"172.21.15.195:6821/821328372","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-17T14:52:34.596386+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-17T14:52:34.584549+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-17T14:52:35.666237+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-17T14:52:39.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-05-17T14:52:39.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-05-17T14:52:39.674 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-05-17T14:52:39.826 INFO:teuthology.orchestra.run.smithi195.stdout:38654705668 2024-05-17T14:52:39.826 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-05-17T14:52:39.827 INFO:teuthology.orchestra.run.smithi195.stdout:38654705667 2024-05-17T14:52:39.828 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-05-17T14:52:39.854 INFO:teuthology.orchestra.run.smithi195.stdout:38654705668 2024-05-17T14:52:39.855 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-05-17T14:52:40.183 INFO:teuthology.orchestra.run.smithi195.stdout:38654705666 2024-05-17T14:52:40.196 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705667 got 38654705666 for osd.1 2024-05-17T14:52:40.199 INFO:teuthology.orchestra.run.smithi195.stdout:38654705667 2024-05-17T14:52:40.213 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705668 got 38654705667 for osd.2 2024-05-17T14:52:40.215 INFO:teuthology.orchestra.run.smithi195.stdout:38654705667 2024-05-17T14:52:40.228 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705668 got 38654705667 for osd.0 2024-05-17T14:52:41.197 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-05-17T14:52:41.213 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-05-17T14:52:41.229 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-05-17T14:52:41.532 INFO:teuthology.orchestra.run.smithi195.stdout:38654705667 2024-05-17T14:52:41.545 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705667 got 38654705667 for osd.1 2024-05-17T14:52:41.545 DEBUG:teuthology.parallel:result is None 2024-05-17T14:52:41.558 INFO:teuthology.orchestra.run.smithi195.stdout:38654705668 2024-05-17T14:52:41.570 INFO:teuthology.orchestra.run.smithi195.stdout:38654705668 2024-05-17T14:52:41.571 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705668 got 38654705668 for osd.2 2024-05-17T14:52:41.571 DEBUG:teuthology.parallel:result is None 2024-05-17T14:52:41.584 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705668 got 38654705668 for osd.0 2024-05-17T14:52:41.584 DEBUG:teuthology.parallel:result is None 2024-05-17T14:52:41.584 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-05-17T14:52:41.584 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-05-17T14:52:41.915 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:41.915 INFO:teuthology.orchestra.run.smithi195.stderr:dumped all 2024-05-17T14:52:41.927 INFO:teuthology.orchestra.run.smithi195.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-05-17T14:52:40.695426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":28,"num_write":41,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":106316,"kb_used_data":288,"kb_used_omap":4,"kb_used_meta":105979,"kb_avail":283009204,"statfs":{"total":289910292480,"available":289801424896,"internally_reserved":0,"allocated":294912,"data_stored":71994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":108522831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.549866"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-17T14:52:39.709531+0000","last_change":"2024-05-17T14:52:38.695559+0000","last_active":"2024-05-17T14:52:39.709531+0000","last_peered":"2024-05-17T14:52:39.709531+0000","last_clean":"2024-05-17T14:52:39.709531+0000","last_became_active":"2024-05-17T14:52:38.695238+0000","last_became_peered":"2024-05-17T14:52:38.695238+0000","last_unstale":"2024-05-17T14:52:39.709531+0000","last_undegraded":"2024-05-17T14:52:39.709531+0000","last_fullsized":"2024-05-17T14:52:39.709531+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_clean_scrub_stamp":"2024-05-17T14:52:37.685708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T17:49:13.906593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-17T14:52:39.709455+0000","last_change":"2024-05-17T14:52:38.697454+0000","last_active":"2024-05-17T14:52:39.709455+0000","last_peered":"2024-05-17T14:52:39.709455+0000","last_clean":"2024-05-17T14:52:39.709455+0000","last_became_active":"2024-05-17T14:52:38.697183+0000","last_became_peered":"2024-05-17T14:52:38.697183+0000","last_unstale":"2024-05-17T14:52:39.709455+0000","last_undegraded":"2024-05-17T14:52:39.709455+0000","last_fullsized":"2024-05-17T14:52:39.709455+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_clean_scrub_stamp":"2024-05-17T14:52:37.685708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T14:53:42.556187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-17T14:52:39.691263+0000","last_change":"2024-05-17T14:52:38.694753+0000","last_active":"2024-05-17T14:52:39.691263+0000","last_peered":"2024-05-17T14:52:39.691263+0000","last_clean":"2024-05-17T14:52:39.691263+0000","last_became_active":"2024-05-17T14:52:38.694165+0000","last_became_peered":"2024-05-17T14:52:38.694165+0000","last_unstale":"2024-05-17T14:52:39.691263+0000","last_undegraded":"2024-05-17T14:52:39.691263+0000","last_fullsized":"2024-05-17T14:52:39.691263+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_clean_scrub_stamp":"2024-05-17T14:52:37.685708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-19T00:47:21.159385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-17T14:52:39.709504+0000","last_change":"2024-05-17T14:52:38.696666+0000","last_active":"2024-05-17T14:52:39.709504+0000","last_peered":"2024-05-17T14:52:39.709504+0000","last_clean":"2024-05-17T14:52:39.709504+0000","last_became_active":"2024-05-17T14:52:38.696358+0000","last_became_peered":"2024-05-17T14:52:38.696358+0000","last_unstale":"2024-05-17T14:52:39.709504+0000","last_undegraded":"2024-05-17T14:52:39.709504+0000","last_fullsized":"2024-05-17T14:52:39.709504+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_clean_scrub_stamp":"2024-05-17T14:52:37.685708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T21:05:40.868133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'19","reported_seq":37,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-17T14:52:39.747674+0000","last_change":"2024-05-17T14:52:39.710679+0000","last_active":"2024-05-17T14:52:39.747674+0000","last_peered":"2024-05-17T14:52:39.747674+0000","last_clean":"2024-05-17T14:52:39.747674+0000","last_became_active":"2024-05-17T14:52:39.710353+0000","last_became_peered":"2024-05-17T14:52:39.710353+0000","last_unstale":"2024-05-17T14:52:39.747674+0000","last_undegraded":"2024-05-17T14:52:39.747674+0000","last_fullsized":"2024-05-17T14:52:39.747674+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:38.686503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:38.686503+0000","last_clean_scrub_stamp":"2024-05-17T14:52:38.686503+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-19T00:25:18.490494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":28,"num_write":41,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-17T14:52:39.709444+0000","last_change":"2024-05-17T14:52:38.695654+0000","last_active":"2024-05-17T14:52:39.709444+0000","last_peered":"2024-05-17T14:52:39.709444+0000","last_clean":"2024-05-17T14:52:39.709444+0000","last_became_active":"2024-05-17T14:52:38.695347+0000","last_became_peered":"2024-05-17T14:52:38.695347+0000","last_unstale":"2024-05-17T14:52:39.709444+0000","last_undegraded":"2024-05-17T14:52:39.709444+0000","last_fullsized":"2024-05-17T14:52:39.709444+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_clean_scrub_stamp":"2024-05-17T14:52:37.685708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T21:52:30.936457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-17T14:52:38.697629+0000","last_change":"2024-05-17T14:52:38.696575+0000","last_active":"2024-05-17T14:52:38.697629+0000","last_peered":"2024-05-17T14:52:38.697629+0000","last_clean":"2024-05-17T14:52:38.697629+0000","last_became_active":"2024-05-17T14:52:38.696222+0000","last_became_peered":"2024-05-17T14:52:38.696222+0000","last_unstale":"2024-05-17T14:52:38.697629+0000","last_undegraded":"2024-05-17T14:52:38.697629+0000","last_fullsized":"2024-05-17T14:52:38.697629+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_clean_scrub_stamp":"2024-05-17T14:52:37.685708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-19T02:23:07.427073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-17T14:52:39.691118+0000","last_change":"2024-05-17T14:52:38.694520+0000","last_active":"2024-05-17T14:52:39.691118+0000","last_peered":"2024-05-17T14:52:39.691118+0000","last_clean":"2024-05-17T14:52:39.691118+0000","last_became_active":"2024-05-17T14:52:38.694235+0000","last_became_peered":"2024-05-17T14:52:38.694235+0000","last_unstale":"2024-05-17T14:52:39.691118+0000","last_undegraded":"2024-05-17T14:52:39.691118+0000","last_fullsized":"2024-05-17T14:52:39.691118+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_clean_scrub_stamp":"2024-05-17T14:52:37.685708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T23:15:21.123642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-17T14:52:39.709571+0000","last_change":"2024-05-17T14:52:38.695522+0000","last_active":"2024-05-17T14:52:39.709571+0000","last_peered":"2024-05-17T14:52:39.709571+0000","last_clean":"2024-05-17T14:52:39.709571+0000","last_became_active":"2024-05-17T14:52:38.695273+0000","last_became_peered":"2024-05-17T14:52:38.695273+0000","last_unstale":"2024-05-17T14:52:39.709571+0000","last_undegraded":"2024-05-17T14:52:39.709571+0000","last_fullsized":"2024-05-17T14:52:39.709571+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_clean_scrub_stamp":"2024-05-17T14:52:37.685708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-19T02:14:20.728562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":28,"num_write":41,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35436,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336404,"statfs":{"total":96636764160,"available":96600477696,"internally_reserved":0,"allocated":98304,"data_stored":23998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35444,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336396,"statfs":{"total":96636764160,"available":96600469504,"internally_reserved":0,"allocated":98304,"data_stored":23998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35436,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336404,"statfs":{"total":96636764160,"available":96600477696,"internally_reserved":0,"allocated":98304,"data_stored":23998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-17T14:52:41.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-05-17T14:52:42.219 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:42.219 INFO:teuthology.orchestra.run.smithi195.stderr:dumped all 2024-05-17T14:52:42.232 INFO:teuthology.orchestra.run.smithi195.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-05-17T14:52:42.146195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":28,"num_write":41,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":107152,"kb_used_data":884,"kb_used_omap":4,"kb_used_meta":105979,"kb_avail":283008368,"statfs":{"total":289910292480,"available":289800568832,"internally_reserved":0,"allocated":905216,"data_stored":669728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":108522831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000635"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:41.205522+0000","last_change":"2024-05-17T14:52:38.695559+0000","last_active":"2024-05-17T14:52:41.205522+0000","last_peered":"2024-05-17T14:52:41.205522+0000","last_clean":"2024-05-17T14:52:41.205522+0000","last_became_active":"2024-05-17T14:52:38.695238+0000","last_became_peered":"2024-05-17T14:52:38.695238+0000","last_unstale":"2024-05-17T14:52:41.205522+0000","last_undegraded":"2024-05-17T14:52:41.205522+0000","last_fullsized":"2024-05-17T14:52:41.205522+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_clean_scrub_stamp":"2024-05-17T14:52:37.685708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T17:49:13.906593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:41.205665+0000","last_change":"2024-05-17T14:52:38.697454+0000","last_active":"2024-05-17T14:52:41.205665+0000","last_peered":"2024-05-17T14:52:41.205665+0000","last_clean":"2024-05-17T14:52:41.205665+0000","last_became_active":"2024-05-17T14:52:38.697183+0000","last_became_peered":"2024-05-17T14:52:38.697183+0000","last_unstale":"2024-05-17T14:52:41.205665+0000","last_undegraded":"2024-05-17T14:52:41.205665+0000","last_fullsized":"2024-05-17T14:52:41.205665+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_clean_scrub_stamp":"2024-05-17T14:52:37.685708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T14:53:42.556187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-17T14:52:39.691263+0000","last_change":"2024-05-17T14:52:38.694753+0000","last_active":"2024-05-17T14:52:39.691263+0000","last_peered":"2024-05-17T14:52:39.691263+0000","last_clean":"2024-05-17T14:52:39.691263+0000","last_became_active":"2024-05-17T14:52:38.694165+0000","last_became_peered":"2024-05-17T14:52:38.694165+0000","last_unstale":"2024-05-17T14:52:39.691263+0000","last_undegraded":"2024-05-17T14:52:39.691263+0000","last_fullsized":"2024-05-17T14:52:39.691263+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_clean_scrub_stamp":"2024-05-17T14:52:37.685708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-19T00:47:21.159385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:41.205635+0000","last_change":"2024-05-17T14:52:38.696666+0000","last_active":"2024-05-17T14:52:41.205635+0000","last_peered":"2024-05-17T14:52:41.205635+0000","last_clean":"2024-05-17T14:52:41.205635+0000","last_became_active":"2024-05-17T14:52:38.696358+0000","last_became_peered":"2024-05-17T14:52:38.696358+0000","last_unstale":"2024-05-17T14:52:41.205635+0000","last_undegraded":"2024-05-17T14:52:41.205635+0000","last_fullsized":"2024-05-17T14:52:41.205635+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_clean_scrub_stamp":"2024-05-17T14:52:37.685708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T21:05:40.868133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'19","reported_seq":37,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-17T14:52:39.747674+0000","last_change":"2024-05-17T14:52:39.710679+0000","last_active":"2024-05-17T14:52:39.747674+0000","last_peered":"2024-05-17T14:52:39.747674+0000","last_clean":"2024-05-17T14:52:39.747674+0000","last_became_active":"2024-05-17T14:52:39.710353+0000","last_became_peered":"2024-05-17T14:52:39.710353+0000","last_unstale":"2024-05-17T14:52:39.747674+0000","last_undegraded":"2024-05-17T14:52:39.747674+0000","last_fullsized":"2024-05-17T14:52:39.747674+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:38.686503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:38.686503+0000","last_clean_scrub_stamp":"2024-05-17T14:52:38.686503+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-19T00:25:18.490494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":28,"num_write":41,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":14,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:41.205519+0000","last_change":"2024-05-17T14:52:38.695654+0000","last_active":"2024-05-17T14:52:41.205519+0000","last_peered":"2024-05-17T14:52:41.205519+0000","last_clean":"2024-05-17T14:52:41.205519+0000","last_became_active":"2024-05-17T14:52:38.695347+0000","last_became_peered":"2024-05-17T14:52:38.695347+0000","last_unstale":"2024-05-17T14:52:41.205519+0000","last_undegraded":"2024-05-17T14:52:41.205519+0000","last_fullsized":"2024-05-17T14:52:41.205519+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_clean_scrub_stamp":"2024-05-17T14:52:37.685708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T21:52:30.936457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-05-17T14:52:38.697629+0000","last_change":"2024-05-17T14:52:38.696575+0000","last_active":"2024-05-17T14:52:38.697629+0000","last_peered":"2024-05-17T14:52:38.697629+0000","last_clean":"2024-05-17T14:52:38.697629+0000","last_became_active":"2024-05-17T14:52:38.696222+0000","last_became_peered":"2024-05-17T14:52:38.696222+0000","last_unstale":"2024-05-17T14:52:38.697629+0000","last_undegraded":"2024-05-17T14:52:38.697629+0000","last_fullsized":"2024-05-17T14:52:38.697629+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_clean_scrub_stamp":"2024-05-17T14:52:37.685708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-19T02:23:07.427073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-05-17T14:52:39.691118+0000","last_change":"2024-05-17T14:52:38.694520+0000","last_active":"2024-05-17T14:52:39.691118+0000","last_peered":"2024-05-17T14:52:39.691118+0000","last_clean":"2024-05-17T14:52:39.691118+0000","last_became_active":"2024-05-17T14:52:38.694235+0000","last_became_peered":"2024-05-17T14:52:38.694235+0000","last_unstale":"2024-05-17T14:52:39.691118+0000","last_undegraded":"2024-05-17T14:52:39.691118+0000","last_fullsized":"2024-05-17T14:52:39.691118+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_clean_scrub_stamp":"2024-05-17T14:52:37.685708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-18T23:15:21.123642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-05-17T14:52:41.205503+0000","last_change":"2024-05-17T14:52:38.695522+0000","last_active":"2024-05-17T14:52:41.205503+0000","last_peered":"2024-05-17T14:52:41.205503+0000","last_clean":"2024-05-17T14:52:41.205503+0000","last_became_active":"2024-05-17T14:52:38.695273+0000","last_became_peered":"2024-05-17T14:52:38.695273+0000","last_unstale":"2024-05-17T14:52:41.205503+0000","last_undegraded":"2024-05-17T14:52:41.205503+0000","last_fullsized":"2024-05-17T14:52:41.205503+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-17T14:52:37.685708+0000","last_clean_scrub_stamp":"2024-05-17T14:52:37.685708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-19T02:14:20.728562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":28,"num_write":41,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":925696,"data_stored":918560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35436,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336404,"statfs":{"total":96636764160,"available":96600477696,"internally_reserved":0,"allocated":98304,"data_stored":23998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36280,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335560,"statfs":{"total":96636764160,"available":96599613440,"internally_reserved":0,"allocated":708608,"data_stored":621732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35436,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336404,"statfs":{"total":96636764160,"available":96600477696,"internally_reserved":0,"allocated":98304,"data_stored":23998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-17T14:52:42.232 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-05-17T14:52:42.233 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-05-17T14:52:42.233 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-05-17T14:52:42.233 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-05-17T14:52:42.560 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-05-17T14:52:42.560 INFO:teuthology.orchestra.run.smithi195.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-17T14:52:42.573 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-05-17T14:52:42.574 INFO:teuthology.run_tasks:Running task exec... 2024-05-17T14:52:42.610 INFO:teuthology.task.exec:Executing custom commands... 2024-05-17T14:52:42.610 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi195.front.sepia.ceph.com 2024-05-17T14:52:42.610 DEBUG:teuthology.orchestra.run.smithi195:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-05-17T14:52:42.984 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror] 2024-05-17T14:52:42.984 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD8bkdmmmZ0DRAAmOKW3JTaaphl9Im2vH3/3w== 2024-05-17T14:52:42.984 INFO:teuthology.orchestra.run.smithi195.stdout: caps mon = "profile rbd-mirror-peer" 2024-05-17T14:52:42.984 INFO:teuthology.orchestra.run.smithi195.stdout: caps osd = "profile rbd" 2024-05-17T14:52:42.985 INFO:teuthology.orchestra.run.smithi195.stderr:updated caps for client.mirror 2024-05-17T14:52:42.998 DEBUG:teuthology.orchestra.run.smithi195:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-17T14:52:43.373 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.0] 2024-05-17T14:52:43.373 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD8bkdmx0bAEBAAidcTnG2fI9jQHMCLDCGlGA== 2024-05-17T14:52:43.373 INFO:teuthology.orchestra.run.smithi195.stdout: caps mon = "profile rbd-mirror" 2024-05-17T14:52:43.374 INFO:teuthology.orchestra.run.smithi195.stdout: caps osd = "profile rbd" 2024-05-17T14:52:43.374 INFO:teuthology.orchestra.run.smithi195.stderr:updated caps for client.mirror.0 2024-05-17T14:52:43.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-17T14:52:43.761 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.1] 2024-05-17T14:52:43.761 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD8bkdm8y1WFhAApbXIdTLv6kZwRjvH37Yx7w== 2024-05-17T14:52:43.761 INFO:teuthology.orchestra.run.smithi195.stdout: caps mon = "profile rbd-mirror" 2024-05-17T14:52:43.762 INFO:teuthology.orchestra.run.smithi195.stdout: caps osd = "profile rbd" 2024-05-17T14:52:43.762 INFO:teuthology.orchestra.run.smithi195.stderr:updated caps for client.mirror.1 2024-05-17T14:52:43.775 DEBUG:teuthology.orchestra.run.smithi195:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-17T14:52:44.154 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.2] 2024-05-17T14:52:44.155 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD8bkdmkwbqGxAAnd6web7SHzC54zR+7AI8OA== 2024-05-17T14:52:44.155 INFO:teuthology.orchestra.run.smithi195.stdout: caps mon = "profile rbd-mirror" 2024-05-17T14:52:44.155 INFO:teuthology.orchestra.run.smithi195.stdout: caps osd = "profile rbd" 2024-05-17T14:52:44.155 INFO:teuthology.orchestra.run.smithi195.stderr:updated caps for client.mirror.2 2024-05-17T14:52:44.168 DEBUG:teuthology.orchestra.run.smithi195:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-17T14:52:44.543 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.3] 2024-05-17T14:52:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQD8bkdmbCt7IRAAYztVN4zwiqXoBJUEnAmM8A== 2024-05-17T14:52:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: caps mon = "profile rbd-mirror" 2024-05-17T14:52:44.543 INFO:teuthology.orchestra.run.smithi195.stdout: caps osd = "profile rbd" 2024-05-17T14:52:44.543 INFO:teuthology.orchestra.run.smithi195.stderr:updated caps for client.mirror.3 2024-05-17T14:52:44.557 DEBUG:teuthology.orchestra.run.smithi195:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-05-17T14:52:44.931 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror] 2024-05-17T14:52:44.931 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAgb0dmUszgFBAATPIA95op0RTQIojqzNOpVQ== 2024-05-17T14:52:44.931 INFO:teuthology.orchestra.run.smithi195.stdout: caps mon = "profile rbd-mirror-peer" 2024-05-17T14:52:44.931 INFO:teuthology.orchestra.run.smithi195.stdout: caps osd = "profile rbd" 2024-05-17T14:52:44.931 INFO:teuthology.orchestra.run.smithi195.stderr:updated caps for client.mirror 2024-05-17T14:52:44.945 DEBUG:teuthology.orchestra.run.smithi195:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-17T14:52:45.316 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.0] 2024-05-17T14:52:45.317 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAgb0dmuXRwGhAAglf+QfVGJBKqwJPfuP1zag== 2024-05-17T14:52:45.317 INFO:teuthology.orchestra.run.smithi195.stdout: caps mon = "profile rbd-mirror" 2024-05-17T14:52:45.317 INFO:teuthology.orchestra.run.smithi195.stdout: caps osd = "profile rbd" 2024-05-17T14:52:45.317 INFO:teuthology.orchestra.run.smithi195.stderr:updated caps for client.mirror.0 2024-05-17T14:52:45.330 DEBUG:teuthology.orchestra.run.smithi195:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-17T14:52:45.699 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.1] 2024-05-17T14:52:45.700 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAgb0dmzWL8HxAA3V2YAxOSIWWCHJwn6O1u6w== 2024-05-17T14:52:45.700 INFO:teuthology.orchestra.run.smithi195.stdout: caps mon = "profile rbd-mirror" 2024-05-17T14:52:45.700 INFO:teuthology.orchestra.run.smithi195.stdout: caps osd = "profile rbd" 2024-05-17T14:52:45.700 INFO:teuthology.orchestra.run.smithi195.stderr:updated caps for client.mirror.1 2024-05-17T14:52:45.713 DEBUG:teuthology.orchestra.run.smithi195:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-17T14:52:46.082 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.2] 2024-05-17T14:52:46.082 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAgb0dmwFiDJRAAxzNXgUmkbVDVUZY0YJZJNA== 2024-05-17T14:52:46.082 INFO:teuthology.orchestra.run.smithi195.stdout: caps mon = "profile rbd-mirror" 2024-05-17T14:52:46.083 INFO:teuthology.orchestra.run.smithi195.stdout: caps osd = "profile rbd" 2024-05-17T14:52:46.083 INFO:teuthology.orchestra.run.smithi195.stderr:updated caps for client.mirror.2 2024-05-17T14:52:46.097 DEBUG:teuthology.orchestra.run.smithi195:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-05-17T14:52:46.468 INFO:teuthology.orchestra.run.smithi195.stdout:[client.mirror.3] 2024-05-17T14:52:46.468 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAgb0dmOIUQKxAAew7c+I8XAd4zoehjsLMl2w== 2024-05-17T14:52:46.468 INFO:teuthology.orchestra.run.smithi195.stdout: caps mon = "profile rbd-mirror" 2024-05-17T14:52:46.468 INFO:teuthology.orchestra.run.smithi195.stdout: caps osd = "profile rbd" 2024-05-17T14:52:46.469 INFO:teuthology.orchestra.run.smithi195.stderr:updated caps for client.mirror.3 2024-05-17T14:52:46.482 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-05-17T14:52:46.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-05-17T14:52:46.493 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-05-17T14:52:46.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-05-17T14:52:46.495 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-05-17T14:52:46.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-05-17T14:52:46.528 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-05-17T14:52:46.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-05-17T14:52:46.530 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-05-17T14:52:46.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-05-17T14:52:46.587 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-05-17T14:52:46.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-05-17T14:52:46.589 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-05-17T14:52:46.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-05-17T14:52:46.635 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-05-17T14:52:46.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-05-17T14:52:46.637 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-05-17T14:52:46.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-05-17T14:52:46.646 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-05-17T14:52:46.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-05-17T14:52:46.648 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-05-17T14:52:46.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-05-17T14:52:46.658 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-05-17T14:52:46.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-05-17T14:52:46.660 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-05-17T14:52:46.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-05-17T14:52:46.670 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-05-17T14:52:46.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-05-17T14:52:46.673 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-05-17T14:52:46.682 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-05-17T14:52:46.682 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-05-17T14:52:46.684 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-05-17T14:52:46.684 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-05-17T14:52:46.696 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1715957566 2024-05-17T14:52:46.696 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-05-17T14:52:46.696 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-05-17T14:52:46.706 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1715957566 2024-05-17T14:52:46.706 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-05-17T14:52:46.706 INFO:teuthology.run_tasks:Running task workunit... 2024-05-17T14:52:46.721 INFO:tasks.workunit:Pulling workunits from ref ee59d86aad04089481e55c1a15aa5871339935b4 2024-05-17T14:52:46.721 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-17T14:52:46.721 DEBUG:teuthology.orchestra.run.smithi195:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-05-17T14:52:46.722 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-05-17T14:52:46.722 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2024-05-17T14:52:46.722 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-05-17T14:52:46.722 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-05-17T14:52:46.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-17T14:52:46.740 INFO:teuthology.orchestra.run.smithi195.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-05-17T14:52:46.741 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-05-17T14:52:46.797 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-05-17T14:52:46.797 DEBUG:teuthology.orchestra.run.smithi195:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-05-17T14:52:46.852 INFO:tasks.workunit:timeout=6h 2024-05-17T14:52:46.853 INFO:tasks.workunit:cleanup=True 2024-05-17T14:52:46.853 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout ee59d86aad04089481e55c1a15aa5871339935b4 2024-05-17T14:52:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-05-17T14:53:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Note: switching to 'ee59d86aad04089481e55c1a15aa5871339935b4'. 2024-05-17T14:53:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:53:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-17T14:53:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-17T14:53:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:state without impacting any branches by switching back to a branch. 2024-05-17T14:53:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:53:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-17T14:53:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-17T14:53:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:53:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: git switch -c 2024-05-17T14:53:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:53:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Or undo this operation with: 2024-05-17T14:53:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:53:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: git switch - 2024-05-17T14:53:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:53:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-17T14:53:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:53:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:HEAD is now at ee59d86aad0 Merge branch 'wip-64800' of https://github.com/ajarr/ceph into HEAD 2024-05-17T14:53:33.672 DEBUG:teuthology.orchestra.run.smithi195:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-05-17T14:53:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-17T14:53:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-05-17T14:53:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-17T14:53:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-17T14:53:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-17T14:53:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-05-17T14:53:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-05-17T14:53:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-17T14:53:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-05-17T14:53:34.587 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-05-17T14:53:34.587 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-05-17T14:53:34.643 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-05-17T14:53:34.645 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-05-17T14:53:34.646 DEBUG:teuthology.orchestra.run.smithi195:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee59d86aad04089481e55c1a15aa5871339935b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-05-17T14:53:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ IMAGE_COUNT=50 2024-05-17T14:53:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ export LOCKDEP=0 2024-05-17T14:53:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ LOCKDEP=0 2024-05-17T14:53:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-05-17T14:53:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-05-17T14:53:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ type xmlstarlet 2024-05-17T14:53:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ XMLSTARLET=xmlstarlet 2024-05-17T14:53:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ RBD_MIRROR_INSTANCES=4 2024-05-17T14:53:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CLUSTER1=cluster1 2024-05-17T14:53:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CLUSTER2=cluster2 2024-05-17T14:53:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ PEER_CLUSTER_SUFFIX= 2024-05-17T14:53:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ POOL=mirror 2024-05-17T14:53:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ PARENT_POOL=mirror_parent 2024-05-17T14:53:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ NS1=ns1 2024-05-17T14:53:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ NS2=ns2 2024-05-17T14:53:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ TEMPDIR= 2024-05-17T14:53:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ID=mirror 2024-05-17T14:53:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-05-17T14:53:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-05-17T14:53:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ RBD_MIRROR_MODE=snapshot 2024-05-17T14:53:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ MIRROR_POOL_MODE=image 2024-05-17T14:53:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' snapshot = snapshot ']' 2024-05-17T14:53:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ MIRROR_POOL_MODE=image 2024-05-17T14:53:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-05-17T14:53:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS='--id mirror' 2024-05-17T14:53:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ LAST_MIRROR_INSTANCE=3 2024-05-17T14:53:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-05-17T14:53:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-05-17T14:53:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-05-17T14:53:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_BIN=. 2024-05-17T14:53:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_SRC=. 2024-05-17T14:53:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' -e CMakeCache.txt ']' 2024-05-17T14:53:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 -gt 0 ']' 2024-05-17T14:53:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ setup 2024-05-17T14:53:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local c 2024-05-17T14:53:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-05-17T14:53:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ setup_tempdir 2024-05-17T14:53:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:53:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mktemp -d 2024-05-17T14:53:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ TEMPDIR=/tmp/tmp.0i1dljubcH 2024-05-17T14:53:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -z 1 ']' 2024-05-17T14:53:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ setup_pools cluster1 cluster2 2024-05-17T14:53:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:53:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local remote_cluster=cluster2 2024-05-17T14:53:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mon_map_file 2024-05-17T14:53:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mon_addr 2024-05-17T14:53:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local admin_key_file 2024-05-17T14:53:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local uuid 2024-05-17T14:53:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T14:53:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-05-17T14:53:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:pool 'mirror' already exists 2024-05-17T14:53:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T14:53:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-05-17T14:53:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:pool 'mirror_parent' already exists 2024-05-17T14:53:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T14:53:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 pool init mirror 2024-05-17T14:53:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T14:53:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-05-17T14:53:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:53:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T14:53:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-05-17T14:53:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-05-17T14:53:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-05-17T14:53:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-05-17T14:53:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-05-17T14:53:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-05-17T14:53:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -z ']' 2024-05-17T14:53:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -z ']' 2024-05-17T14:53:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ peer_add cluster1 mirror cluster2 2024-05-17T14:53:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:53:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:53:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local client_cluster=cluster2 2024-05-17T14:53:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local remote_cluster=cluster2 2024-05-17T14:53:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local uuid_var_name 2024-05-17T14:53:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:53:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local error_code 2024-05-17T14:53:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local peer_uuid 2024-05-17T14:53:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 16 32 2024-05-17T14:53:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T14:53:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-05-17T14:53:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ peer_uuid=19bb688f-aeae-4d87-ba1c-7b8f4695256d 2024-05-17T14:53:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=0 2024-05-17T14:53:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T14:53:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 0 -eq 17 ']' 2024-05-17T14:53:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 0 -eq 0 2024-05-17T14:53:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:53:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:53:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-05-17T14:53:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:53:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror_parent 2024-05-17T14:53:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local client_cluster=cluster2 2024-05-17T14:53:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local remote_cluster=cluster2 2024-05-17T14:53:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local uuid_var_name 2024-05-17T14:53:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:53:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local error_code 2024-05-17T14:53:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local peer_uuid 2024-05-17T14:53:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 16 32 2024-05-17T14:53:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T14:53:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-05-17T14:53:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ peer_uuid=0a5f21f5-898c-4c37-b257-500dcef76f47 2024-05-17T14:53:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=0 2024-05-17T14:53:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T14:53:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 0 -eq 17 ']' 2024-05-17T14:53:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 0 -eq 0 2024-05-17T14:53:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:53:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:53:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ setup_pools cluster2 cluster1 2024-05-17T14:53:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:53:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local remote_cluster=cluster1 2024-05-17T14:53:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mon_map_file 2024-05-17T14:53:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mon_addr 2024-05-17T14:53:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local admin_key_file 2024-05-17T14:53:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local uuid 2024-05-17T14:53:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T14:53:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-05-17T14:53:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:pool 'mirror' already exists 2024-05-17T14:53:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T14:53:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-05-17T14:53:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:pool 'mirror_parent' already exists 2024-05-17T14:53:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T14:53:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 pool init mirror 2024-05-17T14:53:46.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T14:53:46.629+0000 7f297981e640 -1 rbd::mirror::PoolReplayer: 0x563b2ccbec00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-05-17T14:53:46.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T14:53:46.647+0000 7f0c8b21f640 -1 rbd::mirror::PoolReplayer: 0x55e0a3f70c00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-05-17T14:53:46.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T14:53:46.666+0000 7f297981e640 -1 rbd::mirror::PoolReplayer: 0x563b2ceccf00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-05-17T14:53:46.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T14:53:46.690+0000 7f0c8b21f640 -1 rbd::mirror::PoolReplayer: 0x55e0a4c86f00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-05-17T14:53:46.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T14:53:46.705+0000 7f5760a1b640 -1 rbd::mirror::PoolReplayer: 0x55dbbad04c00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-05-17T14:53:46.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T14:53:46.750+0000 7f5760a1b640 -1 rbd::mirror::PoolReplayer: 0x55dbbadd8f00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-05-17T14:53:46.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi195.stderr:2024-05-17T14:53:46.755+0000 7fa8f729f640 -1 rbd::mirror::PoolReplayer: 0x559b9a092c00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-05-17T14:53:46.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi195.stderr:2024-05-17T14:53:46.794+0000 7fa8f729f640 -1 rbd::mirror::PoolReplayer: 0x559b9a184f00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-05-17T14:53:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T14:53:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-05-17T14:53:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:53:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T14:53:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-05-17T14:53:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-05-17T14:53:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-05-17T14:53:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-05-17T14:53:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-05-17T14:53:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-05-17T14:53:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -z ']' 2024-05-17T14:53:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -z ']' 2024-05-17T14:53:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ peer_add cluster2 mirror cluster1 2024-05-17T14:53:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:53:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:53:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local client_cluster=cluster1 2024-05-17T14:53:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local remote_cluster=cluster1 2024-05-17T14:53:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local uuid_var_name 2024-05-17T14:53:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:53:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local error_code 2024-05-17T14:53:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local peer_uuid 2024-05-17T14:53:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 16 32 2024-05-17T14:53:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T14:53:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-05-17T14:53:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ peer_uuid=0605fa7e-6899-4b5e-b894-063138fc8cdf 2024-05-17T14:53:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=0 2024-05-17T14:53:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T14:53:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 0 -eq 17 ']' 2024-05-17T14:53:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 0 -eq 0 2024-05-17T14:53:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:53:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:53:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-05-17T14:53:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:53:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror_parent 2024-05-17T14:53:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local client_cluster=cluster1 2024-05-17T14:53:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local remote_cluster=cluster1 2024-05-17T14:53:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local uuid_var_name 2024-05-17T14:53:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:53:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local error_code 2024-05-17T14:53:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local peer_uuid 2024-05-17T14:53:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 16 32 2024-05-17T14:53:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T14:53:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-05-17T14:53:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ peer_uuid=c49d62a4-e571-4569-9ea7-2f9e326b8d44 2024-05-17T14:53:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=0 2024-05-17T14:53:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T14:53:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 0 -eq 17 ']' 2024-05-17T14:53:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 0 -eq 0 2024-05-17T14:53:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:53:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:53:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:53:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ start_mirrors cluster1 2024-05-17T14:53:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:53:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ seq 0 3 2024-05-17T14:53:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ start_mirror cluster1:0 2024-05-17T14:53:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1:0 2024-05-17T14:53:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local instance 2024-05-17T14:53:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:53:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local val=cluster1:0 2024-05-17T14:53:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster_var_name=cluster 2024-05-17T14:53:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local instance_var_name=instance 2024-05-17T14:53:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cluster=cluster1 2024-05-17T14:53:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ instance=0 2024-05-17T14:53:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 0 = cluster1:0 ']' 2024-05-17T14:53:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ eval cluster=cluster1 2024-05-17T14:53:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:53:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ eval instance=0 2024-05-17T14:53:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:53:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n 1 2024-05-17T14:53:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 2024-05-17T14:53:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ start_mirror cluster1:1 2024-05-17T14:53:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1:1 2024-05-17T14:53:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local instance 2024-05-17T14:53:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-05-17T14:53:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local val=cluster1:1 2024-05-17T14:53:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster_var_name=cluster 2024-05-17T14:53:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local instance_var_name=instance 2024-05-17T14:53:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cluster=cluster1 2024-05-17T14:53:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ instance=1 2024-05-17T14:53:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 1 = cluster1:1 ']' 2024-05-17T14:53:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ eval cluster=cluster1 2024-05-17T14:53:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:53:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ eval instance=1 2024-05-17T14:53:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T14:53:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n 1 2024-05-17T14:53:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 2024-05-17T14:53:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ start_mirror cluster1:2 2024-05-17T14:53:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1:2 2024-05-17T14:53:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local instance 2024-05-17T14:53:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-05-17T14:53:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local val=cluster1:2 2024-05-17T14:53:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster_var_name=cluster 2024-05-17T14:53:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local instance_var_name=instance 2024-05-17T14:53:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cluster=cluster1 2024-05-17T14:53:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ instance=2 2024-05-17T14:53:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 2 = cluster1:2 ']' 2024-05-17T14:53:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ eval cluster=cluster1 2024-05-17T14:53:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:53:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ eval instance=2 2024-05-17T14:53:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T14:53:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n 1 2024-05-17T14:53:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 2024-05-17T14:53:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ start_mirror cluster1:3 2024-05-17T14:53:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1:3 2024-05-17T14:53:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local instance 2024-05-17T14:53:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-05-17T14:53:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local val=cluster1:3 2024-05-17T14:53:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster_var_name=cluster 2024-05-17T14:53:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local instance_var_name=instance 2024-05-17T14:53:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cluster=cluster1 2024-05-17T14:53:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ instance=3 2024-05-17T14:53:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 3 = cluster1:3 ']' 2024-05-17T14:53:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ eval cluster=cluster1 2024-05-17T14:53:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:53:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ eval instance=3 2024-05-17T14:53:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T14:53:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n 1 2024-05-17T14:53:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 2024-05-17T14:53:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ start_mirrors cluster2 2024-05-17T14:53:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:53:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ seq 0 3 2024-05-17T14:53:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ start_mirror cluster2:0 2024-05-17T14:53:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2:0 2024-05-17T14:53:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local instance 2024-05-17T14:53:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-05-17T14:53:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local val=cluster2:0 2024-05-17T14:53:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster_var_name=cluster 2024-05-17T14:53:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local instance_var_name=instance 2024-05-17T14:53:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cluster=cluster2 2024-05-17T14:53:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ instance=0 2024-05-17T14:53:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 0 = cluster2:0 ']' 2024-05-17T14:53:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ eval cluster=cluster2 2024-05-17T14:53:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster2 2024-05-17T14:53:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ eval instance=0 2024-05-17T14:53:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:53:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n 1 2024-05-17T14:53:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 2024-05-17T14:53:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ start_mirror cluster2:1 2024-05-17T14:53:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2:1 2024-05-17T14:53:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local instance 2024-05-17T14:53:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-05-17T14:53:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local val=cluster2:1 2024-05-17T14:53:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster_var_name=cluster 2024-05-17T14:53:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local instance_var_name=instance 2024-05-17T14:53:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cluster=cluster2 2024-05-17T14:53:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ instance=1 2024-05-17T14:53:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 1 = cluster2:1 ']' 2024-05-17T14:53:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ eval cluster=cluster2 2024-05-17T14:53:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster2 2024-05-17T14:53:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ eval instance=1 2024-05-17T14:53:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T14:53:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n 1 2024-05-17T14:53:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 2024-05-17T14:53:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ start_mirror cluster2:2 2024-05-17T14:53:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2:2 2024-05-17T14:53:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local instance 2024-05-17T14:53:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-05-17T14:53:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local val=cluster2:2 2024-05-17T14:53:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster_var_name=cluster 2024-05-17T14:53:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local instance_var_name=instance 2024-05-17T14:53:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cluster=cluster2 2024-05-17T14:53:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ instance=2 2024-05-17T14:53:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 2 = cluster2:2 ']' 2024-05-17T14:53:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ eval cluster=cluster2 2024-05-17T14:53:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster2 2024-05-17T14:53:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ eval instance=2 2024-05-17T14:53:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T14:53:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n 1 2024-05-17T14:53:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 2024-05-17T14:53:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ start_mirror cluster2:3 2024-05-17T14:53:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2:3 2024-05-17T14:53:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local instance 2024-05-17T14:53:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-05-17T14:53:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local val=cluster2:3 2024-05-17T14:53:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster_var_name=cluster 2024-05-17T14:53:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local instance_var_name=instance 2024-05-17T14:53:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cluster=cluster2 2024-05-17T14:53:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ instance=3 2024-05-17T14:53:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 3 = cluster2:3 ']' 2024-05-17T14:53:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ eval cluster=cluster2 2024-05-17T14:53:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster2 2024-05-17T14:53:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ eval instance=3 2024-05-17T14:53:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T14:53:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n 1 2024-05-17T14:53:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 2024-05-17T14:53:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-05-17T14:53:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ tee -a /tmp/tmp.0i1dljubcH/rbd-mirror.test.log 2024-05-17T14:53:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ date '+%F %T' 2024-05-17T14:53:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17 14:53:51 TEST: add image and test replay after client crashes 2024-05-17T14:53:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:2024-05-17 14:53:51 TEST: add image and test replay after client crashes 2024-05-17T14:53:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=test 2024-05-17T14:53:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-05-17T14:53:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:53:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:53:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:53:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T14:53:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T14:53:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror test 512M 2024-05-17T14:53:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:53:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:53:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:53:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T14:53:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 512M ']' 2024-05-17T14:53:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=512M 2024-05-17T14:53:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T14:53:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/test 2024-05-17T14:53:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T14:53:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:53:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T14:53:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:53:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-05-17T14:53:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:53:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:53:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:53:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T14:53:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-05-17T14:53:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T14:53:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-05-17T14:53:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'test': 2024-05-17T14:53:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 512 MiB in 128 objects 2024-05-17T14:53:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T14:53:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T14:53:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10b0bbcfb28a 2024-05-17T14:53:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10b0bbcfb28a 2024-05-17T14:53:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T14:53:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T14:53:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T14:53:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T14:53:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 14:53:51 2024 2024-05-17T14:53:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 14:53:51 2024 2024-05-17T14:53:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 14:53:51 2024 2024-05-17T14:53:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T14:53:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T14:53:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T14:53:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T14:53:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-17T14:53:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:53:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:53:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:53:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-17T14:53:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:53:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:53:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:53:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T14:53:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T14:53:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:53:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:53:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-17T14:53:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:53:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:53:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:53:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T14:53:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T14:53:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T14:53:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-17T14:53:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T14:53:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T14:53:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:53:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T14:53:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T14:53:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T14:53:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:53:53.382 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T14:53:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T14:53:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T14:53:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T14:53:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T14:53:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:53:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:53:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:53:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T14:53:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:53:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:53:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:53:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:53:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T14:53:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:53:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T14:53:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:53:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T14:53:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T14:53:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:53:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:53:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T14:53:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T14:53:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:53:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:53:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:53:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T14:53:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:53:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:53:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T14:53:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T14:53:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T14:53:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:53:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:53:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:53:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T14:53:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:53:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:53:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:53:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T14:53:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T14:53:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:53:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:53:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T14:53:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T14:53:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:53:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:53:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:53:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T14:53:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T14:53:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:53:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T14:53:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:53:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T14:53:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T14:53:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:53:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:53:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T14:53:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T14:53:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:53:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:53:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:53:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T14:53:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:53:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:53:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T14:53:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T14:53:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T14:53:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:53:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:53:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:53:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T14:53:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:53:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:53:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:53:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T14:53:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T14:53:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:53:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:53:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T14:53:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T14:53:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:53:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:53:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:53:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T14:53:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T14:53:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:53:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T14:53:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:53:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T14:53:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T14:53:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:53:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:53:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T14:53:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T14:53:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:53:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:53:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:53:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T14:53:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:53:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:53:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T14:53:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T14:53:53.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T14:53:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:53:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:53:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:53:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T14:53:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:53:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:53:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:53:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-05-17T14:53:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T14:53:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:53:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:53:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T14:53:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T14:53:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:53:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:53:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:53:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T14:53:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T14:53:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:53:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T14:53:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:53:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T14:53:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T14:53:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:53:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:53:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T14:53:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T14:53:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:53:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:53:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:53:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T14:53:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:53:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:53:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T14:53:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T14:53:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T14:53:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:53:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:53:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:53:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-05-17T14:53:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:53:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:53:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:53:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:53:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 1 2024-05-17T14:53:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T14:53:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T14:53:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T14:53:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T14:53:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:53:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:53:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:53:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T14:53:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T14:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T14:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T14:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T14:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:53:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T14:53:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T14:53:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:53:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:53:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:53:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T14:53:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:53:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:53:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T14:53:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T14:53:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T14:53:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:53:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:53:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:53:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T14:53:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:53:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:53:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:53:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T14:53:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T14:53:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:53:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:53:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T14:53:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T14:53:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:53:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:53:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:53:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T14:53:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T14:53:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:53:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T14:53:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:53:54.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T14:53:54.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T14:53:54.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:53:54.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:53:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T14:53:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T14:53:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:53:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:53:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:53:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T14:53:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:53:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:53:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T14:53:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T14:53:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T14:53:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:53:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:53:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:53:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T14:53:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:53:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:53:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:53:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T14:53:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T14:53:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:53:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:53:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T14:53:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T14:53:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:53:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:53:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:53:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T14:53:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T14:53:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:53:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T14:53:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:53:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T14:53:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T14:53:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:53:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:53:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T14:53:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T14:53:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:53:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:53:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:53:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T14:53:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:53:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:53:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T14:53:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T14:53:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T14:53:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:53:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:53:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:53:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T14:53:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:53:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:53:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:53:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-05-17T14:53:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T14:53:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:53:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:53:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T14:53:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T14:53:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:53:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:53:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:53:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T14:53:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T14:53:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:53:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T14:53:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:53:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T14:53:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T14:53:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:53:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:53:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T14:53:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T14:53:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:53:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:53:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:53:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T14:53:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:53:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:53:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T14:53:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T14:53:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T14:53:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:53:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:53:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:53:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-05-17T14:53:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:53:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:53:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:53:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:53:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 2 2024-05-17T14:53:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T14:53:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T14:53:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T14:53:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T14:53:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:53:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:53:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:53:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T14:53:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:53:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:53:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:53:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:53:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T14:53:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:53:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T14:53:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:53:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T14:53:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T14:53:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:53:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:53:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T14:53:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T14:53:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:53:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:53:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:53:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T14:53:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:53:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:53:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T14:53:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T14:53:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T14:53:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:53:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:53:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:53:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T14:53:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:53:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:53:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:53:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T14:53:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T14:53:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:53:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:53:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T14:53:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T14:53:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:53:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:53:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:53:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T14:53:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T14:53:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:53:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T14:53:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:53:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T14:53:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T14:53:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:53:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:53:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T14:53:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T14:53:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:53:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:53:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:53:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T14:53:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:53:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:53:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T14:53:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T14:53:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T14:53:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:53:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:53:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:53:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T14:53:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:53:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:53:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:53:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T14:53:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T14:53:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:53:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:53:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T14:53:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T14:53:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:53:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:53:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:53:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T14:53:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T14:53:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:53:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T14:53:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:53:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T14:53:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T14:53:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:53:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:53:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T14:53:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T14:53:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:53:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:53:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:53:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T14:53:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:53:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:53:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T14:53:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T14:53:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T14:53:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:53:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:53:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:53:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T14:53:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:53:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:53:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:53:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:53:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-05-17T14:53:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T14:53:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:53:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:53:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T14:53:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T14:53:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:53:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:53:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:53:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T14:53:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T14:53:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:53:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T14:53:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:53:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T14:53:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T14:53:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:53:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:53:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T14:53:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T14:53:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:53:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:53:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:53:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:53:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T14:53:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:53:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:53:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T14:53:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T14:53:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T14:53:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:53:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:53:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:53:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-05-17T14:53:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:53:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:53:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:53:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:53:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 4 2024-05-17T14:54:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T14:54:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T14:54:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:54:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T14:54:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T14:54:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:54:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:54:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:54:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T14:54:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:54:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:54:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:54:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:54:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T14:54:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:54:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T14:54:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:54:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T14:54:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T14:54:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:54:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:54:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T14:54:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T14:54:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:54:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:54:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:54:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T14:54:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:54:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:54:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T14:54:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T14:54:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T14:54:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:54:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:54:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:54:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T14:54:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:54:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:54:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:54:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:54:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T14:54:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T14:54:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:54:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:54:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T14:54:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T14:54:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:54:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:54:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:54:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T14:54:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T14:54:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:54:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T14:54:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:54:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T14:54:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T14:54:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:54:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:54:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T14:54:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T14:54:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:54:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:54:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:54:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T14:54:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:54:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:54:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T14:54:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T14:54:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T14:54:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:54:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:54:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:54:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T14:54:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:54:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:54:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:54:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:54:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T14:54:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T14:54:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:54:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:54:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T14:54:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T14:54:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:54:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:54:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:54:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T14:54:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T14:54:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:54:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T14:54:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:54:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T14:54:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T14:54:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:54:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:54:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T14:54:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T14:54:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:54:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:54:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:54:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T14:54:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:54:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:54:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T14:54:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T14:54:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T14:54:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:54:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:54:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:54:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T14:54:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:54:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:54:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:54:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:54:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-05-17T14:54:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T14:54:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:54:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:54:01.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T14:54:01.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T14:54:01.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:54:01.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:54:01.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:54:01.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T14:54:01.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T14:54:01.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:54:01.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:01.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T14:54:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:54:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T14:54:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T14:54:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:54:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:54:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T14:54:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T14:54:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:54:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:54:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:54:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T14:54:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:54:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:54:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T14:54:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T14:54:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T14:54:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:54:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:54:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:54:01.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-05-17T14:54:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:54:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:54:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:54:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:54:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 8 2024-05-17T14:54:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T14:54:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T14:54:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:54:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T14:54:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T14:54:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:54:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:54:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:54:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T14:54:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:54:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:54:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:54:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:54:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T14:54:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:54:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T14:54:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:54:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T14:54:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T14:54:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:54:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:54:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T14:54:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T14:54:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:54:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:54:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:54:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T14:54:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:54:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:54:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T14:54:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T14:54:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T14:54:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:54:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:54:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:54:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T14:54:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:54:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:54:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:54:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T14:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T14:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T14:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T14:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T14:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T14:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T14:54:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:54:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T14:54:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T14:54:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:54:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:54:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T14:54:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T14:54:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:54:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:54:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:54:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T14:54:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:54:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:54:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T14:54:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T14:54:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T14:54:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:54:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:54:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:54:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T14:54:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:54:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:54:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:54:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:54:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T14:54:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T14:54:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:54:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:54:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T14:54:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T14:54:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:54:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T14:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T14:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T14:54:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:54:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T14:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T14:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T14:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T14:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T14:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T14:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T14:54:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T14:54:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:54:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:54:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:54:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T14:54:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:54:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:54:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:54:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:54:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-05-17T14:54:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T14:54:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:54:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:54:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T14:54:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T14:54:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:54:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:54:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:54:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T14:54:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T14:54:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:54:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T14:54:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:54:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T14:54:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T14:54:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:54:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:54:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T14:54:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T14:54:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:54:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:54:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:54:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T14:54:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:54:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:54:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T14:54:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T14:54:10.224 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T14:54:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:54:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:54:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:54:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-05-17T14:54:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:54:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:54:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:54:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:54:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 8 2024-05-17T14:54:15.724 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9628222235537447) > weight (0.25) 2024-05-17T14:54:15.724 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.41209286084543606) > weight (0.25) 2024-05-17T14:54:15.724 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5486037582835176) > weight (0.25) 2024-05-17T14:54:15.724 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7216838915139844) > weight (0.25) 2024-05-17T14:54:15.724 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2024-05-17T14:54:16.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T14:54:16.676+0000 7f297981e640 -1 rbd::mirror::Mirror: 0x563b2c1b2120 update_pool_replayers: restarting failed pool replayer for uuid: 19bb688f-aeae-4d87-ba1c-7b8f4695256d cluster: cluster2 client: client.mirror 2024-05-17T14:54:16.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T14:54:16.697+0000 7f0c8b21f640 -1 rbd::mirror::Mirror: 0x55e0a3474120 update_pool_replayers: restarting failed pool replayer for uuid: 19bb688f-aeae-4d87-ba1c-7b8f4695256d cluster: cluster2 client: client.mirror 2024-05-17T14:54:16.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T14:54:16.724+0000 7f297981e640 -1 rbd::mirror::Mirror: 0x563b2c1b2120 update_pool_replayers: restarting failed pool replayer for uuid: 0a5f21f5-898c-4c37-b257-500dcef76f47 cluster: cluster2 client: client.mirror 2024-05-17T14:54:16.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T14:54:16.750+0000 7f0c8b21f640 -1 rbd::mirror::Mirror: 0x55e0a3474120 update_pool_replayers: restarting failed pool replayer for uuid: 0a5f21f5-898c-4c37-b257-500dcef76f47 cluster: cluster2 client: client.mirror 2024-05-17T14:54:16.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T14:54:16.757+0000 7f5760a1b640 -1 rbd::mirror::Mirror: 0x55dbba0ec120 update_pool_replayers: restarting failed pool replayer for uuid: 19bb688f-aeae-4d87-ba1c-7b8f4695256d cluster: cluster2 client: client.mirror 2024-05-17T14:54:16.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi195.stderr:2024-05-17T14:54:16.802+0000 7fa8f729f640 -1 rbd::mirror::Mirror: 0x559b9947c120 update_pool_replayers: restarting failed pool replayer for uuid: 19bb688f-aeae-4d87-ba1c-7b8f4695256d cluster: cluster2 client: client.mirror 2024-05-17T14:54:16.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T14:54:16.833+0000 7f5760a1b640 -1 rbd::mirror::Mirror: 0x55dbba0ec120 update_pool_replayers: restarting failed pool replayer for uuid: 0a5f21f5-898c-4c37-b257-500dcef76f47 cluster: cluster2 client: client.mirror 2024-05-17T14:54:16.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi195.stderr:2024-05-17T14:54:16.897+0000 7fa8f729f640 -1 rbd::mirror::Mirror: 0x559b9947c120 update_pool_replayers: restarting failed pool replayer for uuid: 0a5f21f5-898c-4c37-b257-500dcef76f47 cluster: cluster2 client: client.mirror 2024-05-17T14:54:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T14:54:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T14:54:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:54:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T14:54:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T14:54:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:54:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:54:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:54:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T14:54:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:54:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:54:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:54:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:54:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T14:54:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:54:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T14:54:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:54:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T14:54:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T14:54:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:54:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:54:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T14:54:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T14:54:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:54:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:54:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:54:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T14:54:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:54:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:54:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T14:54:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T14:54:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T14:54:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:54:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:54:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:54:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T14:54:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:54:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:54:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:54:18.422 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:54:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T14:54:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T14:54:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:54:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:54:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T14:54:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T14:54:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:54:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:54:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:54:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T14:54:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T14:54:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:54:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T14:54:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:54:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T14:54:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T14:54:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:54:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:54:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T14:54:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T14:54:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:54:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:54:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:54:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T14:54:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:54:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:54:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T14:54:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T14:54:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T14:54:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:54:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:54:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:54:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T14:54:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:54:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:54:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:54:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:54:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T14:54:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T14:54:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:54:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:54:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T14:54:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T14:54:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:54:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:54:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:54:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T14:54:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T14:54:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:54:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T14:54:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:54:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T14:54:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T14:54:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:54:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:54:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T14:54:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T14:54:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:54:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:54:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:54:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T14:54:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:54:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:54:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T14:54:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T14:54:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T14:54:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:54:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:54:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:54:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T14:54:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:54:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:54:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:54:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:54:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-05-17T14:54:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T14:54:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:54:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:54:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T14:54:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T14:54:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:54:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:54:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:54:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T14:54:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T14:54:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:54:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T14:54:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:54:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T14:54:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T14:54:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:54:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:54:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T14:54:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T14:54:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:54:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:54:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:54:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T14:54:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:54:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:54:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T14:54:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T14:54:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T14:54:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:54:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:54:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:54:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-05-17T14:54:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:54:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:54:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:54:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:54:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 8 2024-05-17T14:54:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T14:54:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T14:54:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:54:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T14:54:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T14:54:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:54:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:54:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:54:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T14:54:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:54:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:54:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:54:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:54:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T14:54:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:54:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T14:54:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:54:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T14:54:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T14:54:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:54:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:54:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T14:54:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T14:54:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:54:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:54:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:54:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:54:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T14:54:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:54:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:54:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T14:54:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T14:54:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T14:54:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:54:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:54:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:54:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T14:54:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T14:54:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T14:54:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T14:54:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T14:54:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T14:54:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T14:54:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T14:54:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:54:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ clean_snap_name= 2024-05-17T14:54:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ seq 1 10 2024-05-17T14:54:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T14:54:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror test 2024-05-17T14:54:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:54:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:54:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:54:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T14:54:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=9 2024-05-17T14:54:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T14:54:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-17T14:54:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 27572224 bytes 2024-05-17T14:54:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 85614592 bytes 2024-05-17T14:54:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 127092736 bytes 2024-05-17T14:54:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 162726912 bytes 2024-05-17T14:54:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 237112320 bytes 2024-05-17T14:54:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 314274816 bytes 2024-05-17T14:54:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 335668224 bytes 2024-05-17T14:54:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 336292864 bytes 2024-05-17T14:54:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 377261056 bytes 2024-05-17T14:54:28.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 415941632 bytes 2024-05-17T14:54:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 449837056 bytes 2024-05-17T14:54:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 488359936 bytes 2024-05-17T14:54:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 499596288 bytes 2024-05-17T14:54:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 584407040 bytes 2024-05-17T14:54:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 585017344 bytes 2024-05-17T14:54:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 639347712 bytes 2024-05-17T14:54:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 670246912 bytes 2024-05-17T14:54:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 747281408 bytes 2024-05-17T14:54:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 805766144 bytes 2024-05-17T14:54:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 836085760 bytes 2024-05-17T14:54:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 848519168 bytes 2024-05-17T14:54:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 918654976 bytes 2024-05-17T14:54:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 921954304 bytes 2024-05-17T14:54:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 927928320 bytes 2024-05-17T14:54:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 985997312 bytes 2024-05-17T14:54:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1015986176 bytes 2024-05-17T14:54:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1096227840 bytes 2024-05-17T14:54:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1128472576 bytes 2024-05-17T14:54:29.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1158487040 bytes 2024-05-17T14:54:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1178681344 bytes 2024-05-17T14:54:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1192491008 bytes 2024-05-17T14:54:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1253561344 bytes 2024-05-17T14:54:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1292560384 bytes 2024-05-17T14:54:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1319882752 bytes 2024-05-17T14:54:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1343305728 bytes 2024-05-17T14:54:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1359609856 bytes 2024-05-17T14:54:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1406973952 bytes 2024-05-17T14:54:30.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1442042880 bytes 2024-05-17T14:54:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1474896896 bytes 2024-05-17T14:54:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1502172160 bytes 2024-05-17T14:54:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1567314944 bytes 2024-05-17T14:54:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1631917056 bytes 2024-05-17T14:54:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1702288384 bytes 2024-05-17T14:54:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1736073216 bytes 2024-05-17T14:54:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1792262144 bytes 2024-05-17T14:54:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1811213312 bytes 2024-05-17T14:54:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1901769728 bytes 2024-05-17T14:54:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2053596160 bytes 2024-05-17T14:54:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2100439040 bytes 2024-05-17T14:54:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2128940032 bytes 2024-05-17T14:54:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2157106176 bytes 2024-05-17T14:54:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2208779264 bytes 2024-05-17T14:54:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2272473088 bytes 2024-05-17T14:54:32.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2309662720 bytes 2024-05-17T14:54:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2363640832 bytes 2024-05-17T14:54:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2468194304 bytes 2024-05-17T14:54:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2538442752 bytes 2024-05-17T14:54:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2602713088 bytes 2024-05-17T14:54:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2657925120 bytes 2024-05-17T14:54:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2707279872 bytes 2024-05-17T14:54:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2757580800 bytes 2024-05-17T14:54:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2832546816 bytes 2024-05-17T14:54:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2848550912 bytes 2024-05-17T14:54:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2849181696 bytes 2024-05-17T14:54:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2896925696 bytes 2024-05-17T14:54:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2980834304 bytes 2024-05-17T14:54:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 3056879616 bytes 2024-05-17T14:54:33.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 3063047168 bytes 2024-05-17T14:54:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 3090778112 bytes 2024-05-17T14:54:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 3156869120 bytes 2024-05-17T14:54:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3200243712 bytes 2024-05-17T14:54:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3242438656 bytes 2024-05-17T14:54:34.256 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3275115520 bytes 2024-05-17T14:54:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3321607168 bytes 2024-05-17T14:54:35.323 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3325268992 bytes 2024-05-17T14:54:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T14:54:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T14:54:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T14:54:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:54:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-17T14:54:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:54:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:54:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:54:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:54:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T14:54:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T14:54:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:54:35.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:54:36.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-17T14:54:36.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:54:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:54:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:54:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:54:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T14:54:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T14:54:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T14:54:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T14:54:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-17T14:54:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='test: 2024-05-17T14:54:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: global_id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T14:54:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: down+unknown 2024-05-17T14:54:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: status not found 2024-05-17T14:54:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17T14:54:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: peer_sites: 2024-05-17T14:54:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: name: cluster2 2024-05-17T14:54:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: down+unknown 2024-05-17T14:54:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: status not found 2024-05-17T14:54:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: ' 2024-05-17T14:54:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ state: .*up+replaying' 2024-05-17T14:54:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 1 2024-05-17T14:54:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:54:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 2 2024-05-17T14:54:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-17T14:54:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:54:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:54:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:54:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:54:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T14:54:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T14:54:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T14:54:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T14:54:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-17T14:54:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='test: 2024-05-17T14:54:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: global_id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T14:54:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: down+unknown 2024-05-17T14:54:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: status not found 2024-05-17T14:54:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17T14:54:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: peer_sites: 2024-05-17T14:54:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: name: cluster2 2024-05-17T14:54:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: down+unknown 2024-05-17T14:54:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: status not found 2024-05-17T14:54:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: ' 2024-05-17T14:54:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ state: .*up+replaying' 2024-05-17T14:54:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 1 2024-05-17T14:54:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:54:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 4 2024-05-17T14:54:40.723 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8867151749283121) > weight (0.25) 2024-05-17T14:54:40.724 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.2762497474478095) > weight (0.25) 2024-05-17T14:54:40.724 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6191249486445027) > weight (0.25) 2024-05-17T14:54:40.724 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9769302246192804) > weight (0.25) 2024-05-17T14:54:40.724 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2024-05-17T14:54:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-17T14:54:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:54:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:54:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:54:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:54:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T14:54:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T14:54:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T14:54:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T14:54:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-17T14:54:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='test: 2024-05-17T14:54:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: global_id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T14:54:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: down+unknown 2024-05-17T14:54:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: status not found 2024-05-17T14:54:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17T14:54:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: peer_sites: 2024-05-17T14:54:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: name: cluster2 2024-05-17T14:54:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: down+unknown 2024-05-17T14:54:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: status not found 2024-05-17T14:54:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: ' 2024-05-17T14:54:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ state: .*up+replaying' 2024-05-17T14:54:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 1 2024-05-17T14:54:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:54:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 8 2024-05-17T14:54:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-17T14:54:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:54:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:54:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:54:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:54:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T14:54:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T14:54:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T14:54:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T14:54:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-17T14:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='test: 2024-05-17T14:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: global_id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T14:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+replaying 2024-05-17T14:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715957632,"remote_snapshot_timestamp":1715957632,"replay_state":"idle"} 2024-05-17T14:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: service: mirror.0 on smithi195 2024-05-17T14:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 14:54:46 2024-05-17T14:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: peer_sites: 2024-05-17T14:54:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: name: cluster2 2024-05-17T14:54:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+stopped 2024-05-17T14:54:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: local image is primary 2024-05-17T14:54:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 14:54:46' 2024-05-17T14:54:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ state: .*up+replaying' 2024-05-17T14:54:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T14:54:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ description: .*' 2024-05-17T14:54:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715957632,"remote_snapshot_timestamp":1715957632,"replay_state":"idle"} 2024-05-17T14:54:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:54:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T14:54:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^up+' 2024-05-17T14:54:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T14:54:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep 'service: *mirror..* on ' 2024-05-17T14:54:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.0 on smithi195 2024-05-17T14:54:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update 2024-05-17T14:54:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-17T14:54:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 14:54:46' 2024-05-17T14:54:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-17 14:54:46' 2024-05-17T14:54:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:54:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:54:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:54:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:54:51.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local 'prev_last_update=2024-05-17 14:54:46' 2024-05-17T14:54:51.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T14:54:51.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T14:54:51.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-17T14:54:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='OKOKOK142114285mirror.0smithi19519.0.0-3735-gee59d86atrueOK42144288mirror.1smithi19519.0.0-3735-gee59d86afalseOK42184296mirror.2smithi19519.0.0-3735-gee59d86afalseOK42224307mirror.3smithi19519.0.0-3735-gee59d86afalseOKtest72894289-d917-4c92-922e-8f6ade3b96caup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715957632,"remote_snapshot_timestamp":1715957632,"replay_state":"idle"}42114285mirror.0smithi1952024-05-17 14:54:46cluster27e816122-c11e-4a54-9bab-7372f406213bup+stoppedlocal image is primary2024-05-17 14:54:46' 2024-05-17T14:54:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update state 2024-05-17T14:54:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-17T14:54:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 14:54:46' 2024-05-17T14:54:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-17T14:54:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ state=up+replaying 2024-05-17T14:54:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T14:54:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep up+replaying 2024-05-17T14:54:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T14:54:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:54:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:54:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap1 2024-05-17T14:54:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap1 2024-05-17T14:54:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:54:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:54:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:54:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap1 2024-05-17T14:54:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-05-17T14:55:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-17T14:55:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:55:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-17T14:55:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:55:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T14:55:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T14:55:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:55:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:55:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-17T14:55:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:55:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T14:55:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T14:55:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T14:55:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-17T14:55:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T14:55:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:55:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T14:55:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T14:55:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T14:55:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:55:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T14:55:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T14:55:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T14:55:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T14:55:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:55:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T14:55:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T14:55:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:55:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:55:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:55:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T14:55:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:55:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:55:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:55:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:55:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T14:55:04.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:55:04.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:55:04.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T14:55:04.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:55:04.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T14:55:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T14:55:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:55:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:55:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T14:55:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T14:55:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:55:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:55:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:55:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:55:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T14:55:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:55:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:55:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T14:55:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T14:55:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T14:55:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:55:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:55:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:55:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T14:55:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T14:55:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T14:55:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T14:55:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T14:55:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T14:55:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T14:55:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T14:55:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:55:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-17T14:55:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T14:55:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:55:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T14:55:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T14:55:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-05-17T14:55:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T14:55:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:55:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-test.status 2024-05-17T14:55:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-test.status 2024-05-17T14:55:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T14:55:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:55:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-test.status 2024-05-17T14:55:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-test.status 2024-05-17T14:55:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T14:55:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:55:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-05-17T14:55:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:55:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-05-17T14:55:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 5 2024-05-17T14:55:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:55:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:55:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:55:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-05-17T14:55:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:55:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T14:55:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=5 2024-05-17T14:55:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:55:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:55:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T14:55:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:55:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:55:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:55:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:55:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:55:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:55:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=3 2024-05-17T14:55:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 5 = 3 2024-05-17T14:55:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:55:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T14:55:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:55:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:55:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:55:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:55:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:55:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:55:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:55:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T14:55:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:55:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T14:55:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T14:55:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 5 = '' 2024-05-17T14:55:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:55:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T14:55:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:55:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:55:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:55:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:55:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:55:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:55:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:55:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T14:55:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:55:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T14:55:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T14:55:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 5 = '' 2024-05-17T14:55:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:55:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T14:55:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:55:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:55:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:55:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:55:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:55:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:55:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=5 2024-05-17T14:55:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 5 = 5 2024-05-17T14:55:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:55:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-05-17T14:55:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:55:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap1 2024-05-17T14:55:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T14:55:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T14:55:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:55:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-05-17T14:55:09.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'test': 2024-05-17T14:55:09.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 512 MiB in 128 objects 2024-05-17T14:55:09.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T14:55:09.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T14:55:09.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10bd697942f 2024-05-17T14:55:09.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10bd697942f 2024-05-17T14:55:09.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T14:55:09.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T14:55:09.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T14:55:09.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T14:55:09.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 14:54:18 2024 2024-05-17T14:55:09.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 14:54:18 2024 2024-05-17T14:55:09.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 14:54:18 2024 2024-05-17T14:55:09.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T14:55:09.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T14:55:09.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T14:55:09.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T14:55:09.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T14:55:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:55:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:55:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap1 2024-05-17T14:55:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:09.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap1 2024-05-17T14:55:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T14:55:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:55:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:55:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:55:09.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:55:10.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:55:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:55:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:55:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:55:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:55:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:55:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ clean_snap_name=snap1-clean 2024-05-17T14:55:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-05-17T14:55:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:55:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap1-clean 2024-05-17T14:55:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-05-17T14:55:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T14:55:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror test 2024-05-17T14:55:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:55:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T14:55:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=12 2024-05-17T14:55:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T14:55:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-17T14:55:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 17603584 bytes 2024-05-17T14:55:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 53414912 bytes 2024-05-17T14:55:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 98946048 bytes 2024-05-17T14:55:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 135858176 bytes 2024-05-17T14:55:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 219653120 bytes 2024-05-17T14:55:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 272808960 bytes 2024-05-17T14:55:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 315031552 bytes 2024-05-17T14:55:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 327121920 bytes 2024-05-17T14:55:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 375974912 bytes 2024-05-17T14:55:13.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 439728128 bytes 2024-05-17T14:55:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 460895232 bytes 2024-05-17T14:55:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 504872960 bytes 2024-05-17T14:55:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 521698304 bytes 2024-05-17T14:55:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 584489984 bytes 2024-05-17T14:55:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 656837632 bytes 2024-05-17T14:55:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 657456128 bytes 2024-05-17T14:55:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 719948800 bytes 2024-05-17T14:55:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 753867776 bytes 2024-05-17T14:55:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 787716096 bytes 2024-05-17T14:55:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 796731392 bytes 2024-05-17T14:55:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 819048448 bytes 2024-05-17T14:55:14.470 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 840361984 bytes 2024-05-17T14:55:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 943520768 bytes 2024-05-17T14:55:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 950148096 bytes 2024-05-17T14:55:14.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 977238016 bytes 2024-05-17T14:55:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1011235840 bytes 2024-05-17T14:55:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1092593664 bytes 2024-05-17T14:55:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1131004928 bytes 2024-05-17T14:55:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1197973504 bytes 2024-05-17T14:55:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1240460288 bytes 2024-05-17T14:55:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1322688512 bytes 2024-05-17T14:55:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1353812992 bytes 2024-05-17T14:55:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1355173888 bytes 2024-05-17T14:55:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1404830720 bytes 2024-05-17T14:55:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1459762176 bytes 2024-05-17T14:55:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1508747264 bytes 2024-05-17T14:55:16.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1579606016 bytes 2024-05-17T14:55:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1625609216 bytes 2024-05-17T14:55:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1642112000 bytes 2024-05-17T14:55:16.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1658634240 bytes 2024-05-17T14:55:16.379 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1688379392 bytes 2024-05-17T14:55:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1729103872 bytes 2024-05-17T14:55:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1747478528 bytes 2024-05-17T14:55:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1815685120 bytes 2024-05-17T14:55:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1816307712 bytes 2024-05-17T14:55:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1863153664 bytes 2024-05-17T14:55:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1930110976 bytes 2024-05-17T14:55:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 1932764160 bytes 2024-05-17T14:55:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 1971654656 bytes 2024-05-17T14:55:17.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2052377600 bytes 2024-05-17T14:55:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2076558336 bytes 2024-05-17T14:55:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2129545216 bytes 2024-05-17T14:55:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2175140864 bytes 2024-05-17T14:55:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2233348096 bytes 2024-05-17T14:55:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2247400448 bytes 2024-05-17T14:55:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2309839872 bytes 2024-05-17T14:55:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2366081024 bytes 2024-05-17T14:55:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2409824256 bytes 2024-05-17T14:55:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2459274240 bytes 2024-05-17T14:55:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2545003520 bytes 2024-05-17T14:55:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2565910528 bytes 2024-05-17T14:55:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2572980224 bytes 2024-05-17T14:55:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2619682816 bytes 2024-05-17T14:55:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2655392768 bytes 2024-05-17T14:55:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2685598720 bytes 2024-05-17T14:55:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2735988736 bytes 2024-05-17T14:55:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2798270464 bytes 2024-05-17T14:55:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2869857280 bytes 2024-05-17T14:55:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2920380416 bytes 2024-05-17T14:55:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 2977829888 bytes 2024-05-17T14:55:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3012745216 bytes 2024-05-17T14:55:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3069527040 bytes 2024-05-17T14:55:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3099936768 bytes 2024-05-17T14:55:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3115261952 bytes 2024-05-17T14:55:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3177580544 bytes 2024-05-17T14:55:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3237833728 bytes 2024-05-17T14:55:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3287413760 bytes 2024-05-17T14:55:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3291301888 bytes 2024-05-17T14:55:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3350582272 bytes 2024-05-17T14:55:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3401162752 bytes 2024-05-17T14:55:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3446832128 bytes 2024-05-17T14:55:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3493369856 bytes 2024-05-17T14:55:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3519772672 bytes 2024-05-17T14:55:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3564325888 bytes 2024-05-17T14:55:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3603798016 bytes 2024-05-17T14:55:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3617439744 bytes 2024-05-17T14:55:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3655000064 bytes 2024-05-17T14:55:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3668192256 bytes 2024-05-17T14:55:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3673911296 bytes 2024-05-17T14:55:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3694941184 bytes 2024-05-17T14:55:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 3725316096 bytes 2024-05-17T14:55:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9200 IOs, 3772795904 bytes 2024-05-17T14:55:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9300 IOs, 3833672704 bytes 2024-05-17T14:55:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9400 IOs, 3834570752 bytes 2024-05-17T14:55:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9500 IOs, 3840523264 bytes 2024-05-17T14:55:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9600 IOs, 3858468864 bytes 2024-05-17T14:55:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9700 IOs, 3903933440 bytes 2024-05-17T14:55:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9800 IOs, 3904533504 bytes 2024-05-17T14:55:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9900 IOs, 3929857024 bytes 2024-05-17T14:55:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10000 IOs, 4020481024 bytes 2024-05-17T14:55:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10100 IOs, 4078949376 bytes 2024-05-17T14:55:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10200 IOs, 4140780544 bytes 2024-05-17T14:55:22.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10300 IOs, 4203057152 bytes 2024-05-17T14:55:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10400 IOs, 4213611520 bytes 2024-05-17T14:55:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10500 IOs, 4236551168 bytes 2024-05-17T14:55:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10600 IOs, 4256830464 bytes 2024-05-17T14:55:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10700 IOs, 4299679744 bytes 2024-05-17T14:55:22.777 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10800 IOs, 4336094208 bytes 2024-05-17T14:55:22.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10900 IOs, 4375780352 bytes 2024-05-17T14:55:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11000 IOs, 4404519936 bytes 2024-05-17T14:55:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11100 IOs, 4430960640 bytes 2024-05-17T14:55:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11200 IOs, 4505717760 bytes 2024-05-17T14:55:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11300 IOs, 4506351616 bytes 2024-05-17T14:55:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11400 IOs, 4608148480 bytes 2024-05-17T14:55:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11500 IOs, 4650765312 bytes 2024-05-17T14:55:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11600 IOs, 4678471680 bytes 2024-05-17T14:55:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11700 IOs, 4727452672 bytes 2024-05-17T14:55:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11800 IOs, 4749017088 bytes 2024-05-17T14:55:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11900 IOs, 4797374464 bytes 2024-05-17T14:55:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:12000 IOs, 4858450944 bytes 2024-05-17T14:55:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:12100 IOs, 4892381184 bytes 2024-05-17T14:55:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:12200 IOs, 4942507008 bytes 2024-05-17T14:55:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:12300 IOs, 4950560768 bytes 2024-05-17T14:55:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T14:55:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T14:55:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T14:55:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:55:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-17T14:55:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:55:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:24.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:55:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T14:55:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T14:55:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:55:24.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:55:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-17T14:55:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:55:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:55:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T14:55:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T14:55:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T14:55:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T14:55:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-17T14:55:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='test: 2024-05-17T14:55:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: global_id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T14:55:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+replaying 2024-05-17T14:55:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: replaying, {"bytes_per_second":17862826.67,"bytes_per_snapshot":267942400.0,"last_snapshot_bytes":535884800,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957705,"remote_snapshot_timestamp":1715957705,"replay_state":"idle"} 2024-05-17T14:55:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: service: mirror.0 on smithi195 2024-05-17T14:55:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 14:55:16 2024-05-17T14:55:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: peer_sites: 2024-05-17T14:55:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: name: cluster2 2024-05-17T14:55:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+stopped 2024-05-17T14:55:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: local image is primary 2024-05-17T14:55:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 14:55:16' 2024-05-17T14:55:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ state: .*up+replaying' 2024-05-17T14:55:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T14:55:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ description: .*' 2024-05-17T14:55:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":17862826.67,"bytes_per_snapshot":267942400.0,"last_snapshot_bytes":535884800,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957705,"remote_snapshot_timestamp":1715957705,"replay_state":"idle"} 2024-05-17T14:55:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:55:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T14:55:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^up+' 2024-05-17T14:55:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T14:55:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep 'service: *mirror..* on ' 2024-05-17T14:55:25.374 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.0 on smithi195 2024-05-17T14:55:25.374 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update 2024-05-17T14:55:25.374 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-17T14:55:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 14:55:16' 2024-05-17T14:55:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-17 14:55:16' 2024-05-17T14:55:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:55:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:55:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local 'prev_last_update=2024-05-17 14:55:16' 2024-05-17T14:55:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T14:55:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T14:55:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-17T14:55:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='OKOKOK142114285mirror.0smithi19519.0.0-3735-gee59d86atrueOK42144288mirror.1smithi19519.0.0-3735-gee59d86afalseOK42184296mirror.2smithi19519.0.0-3735-gee59d86afalseOK42224307mirror.3smithi19519.0.0-3735-gee59d86afalseOKtest72894289-d917-4c92-922e-8f6ade3b96caup+replayingreplaying, {"bytes_per_second":17862826.67,"bytes_per_snapshot":267942400.0,"last_snapshot_bytes":535884800,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957705,"remote_snapshot_timestamp":1715957705,"replay_state":"idle"}42114285mirror.0smithi1952024-05-17 14:55:16cluster27e816122-c11e-4a54-9bab-7372f406213bup+stoppedlocal image is primary2024-05-17 14:55:16' 2024-05-17T14:55:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update state 2024-05-17T14:55:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-17T14:55:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 14:55:16' 2024-05-17T14:55:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-17T14:55:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ state=up+replaying 2024-05-17T14:55:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T14:55:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep up+replaying 2024-05-17T14:55:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T14:55:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:55:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:55:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap2 2024-05-17T14:55:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap2 2024-05-17T14:55:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:55:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap2 2024-05-17T14:55:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-05-17T14:55:43.725 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5108791313445922) > weight (0.25) 2024-05-17T14:55:43.726 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-05-17T14:55:43.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-05-17T14:55:43.727 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-05-17T14:55:43.727 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-05-17T14:55:43.727 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T14:55:43.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi195.stderr:2024-05-17T14:55:43.727+0000 7f939dea9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 38126) UID: 1000 2024-05-17T14:55:49.830 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-05-17T14:55:49.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-05-17T14:55:49.830 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-05-17T14:55:49.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-05-17T14:55:49.833 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2024-05-17T14:55:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-17T14:55:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:55:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-17T14:55:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:55:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T14:55:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T14:55:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:55:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:55:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-17T14:55:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:55:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T14:55:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T14:55:55.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T14:55:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-17T14:55:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T14:55:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T14:55:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:55:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T14:55:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T14:55:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T14:55:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:55:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T14:55:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T14:55:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T14:55:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:55:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T14:55:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T14:55:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:55:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:55:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:55:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T14:55:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:55:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:55:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:55:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:55:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T14:55:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:55:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:55:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T14:55:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:55:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T14:55:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T14:55:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:55:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:55:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T14:55:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T14:55:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:55:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:55:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:55:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:55:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T14:55:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:55:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:55:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T14:55:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T14:55:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T14:55:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:55:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:55:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:55:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T14:55:55.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T14:55:55.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T14:55:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T14:55:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T14:55:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T14:55:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T14:55:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T14:55:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:55:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-17T14:55:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T14:55:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:55:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T14:55:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T14:55:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-05-17T14:55:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T14:55:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:55:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-test.status 2024-05-17T14:55:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-test.status 2024-05-17T14:55:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T14:55:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:55:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-test.status 2024-05-17T14:55:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-test.status 2024-05-17T14:55:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T14:55:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:55:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-05-17T14:55:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:55:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-05-17T14:55:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 8 2024-05-17T14:55:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:55:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:55:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:55:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-05-17T14:55:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:55:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T14:55:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=8 2024-05-17T14:55:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:55:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:55:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T14:55:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:55:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:55:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:55:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:55:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:55:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:55:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:55:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T14:55:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:55:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T14:55:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T14:55:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 8 = '' 2024-05-17T14:55:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:55:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T14:55:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:55:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:55:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:55:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:55:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:55:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:55:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:55:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T14:55:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:55:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T14:55:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T14:55:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 8 = '' 2024-05-17T14:55:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:55:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T14:55:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:55:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:55:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:55:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:55:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:55:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:55:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:55:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:55:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:55:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T14:55:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:55:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T14:55:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T14:55:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 8 = '' 2024-05-17T14:55:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:55:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T14:56:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:56:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:56:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:56:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:56:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:56:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:56:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:56:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:56:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=8 2024-05-17T14:56:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 8 = 8 2024-05-17T14:56:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:56:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-05-17T14:56:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:56:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:56:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:56:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap2 2024-05-17T14:56:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T14:56:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T14:56:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:56:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-05-17T14:56:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'test': 2024-05-17T14:56:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 512 MiB in 128 objects 2024-05-17T14:56:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T14:56:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 4 2024-05-17T14:56:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10bd697942f 2024-05-17T14:56:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10bd697942f 2024-05-17T14:56:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T14:56:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T14:56:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T14:56:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T14:56:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 14:54:18 2024 2024-05-17T14:56:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 14:54:18 2024 2024-05-17T14:56:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 14:54:18 2024 2024-05-17T14:56:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T14:56:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T14:56:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T14:56:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T14:56:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T14:56:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:56:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snap1-clean ']' 2024-05-17T14:56:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap1-clean 2024-05-17T14:56:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:56:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:56:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap1-clean 2024-05-17T14:56:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T14:56:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:56:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:56:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:56:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:56:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:56:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:56:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:56:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:56:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:56:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:56:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap2 2024-05-17T14:56:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:56:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:56:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap2 2024-05-17T14:56:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T14:56:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:56:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:56:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:56:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:56:04.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:56:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:56:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:56:04.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:56:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:56:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:56:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ clean_snap_name=snap2-clean 2024-05-17T14:56:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-05-17T14:56:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:56:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:56:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:56:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap2-clean 2024-05-17T14:56:05.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-05-17T14:56:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T14:56:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror test 2024-05-17T14:56:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:56:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:56:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:56:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T14:56:05.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=11 2024-05-17T14:56:05.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T14:56:05.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-17T14:56:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 97457152 bytes 2024-05-17T14:56:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 157252608 bytes 2024-05-17T14:56:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 247114752 bytes 2024-05-17T14:56:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 303825920 bytes 2024-05-17T14:56:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 344516608 bytes 2024-05-17T14:56:06.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 408897536 bytes 2024-05-17T14:56:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 431563776 bytes 2024-05-17T14:56:06.832 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 479090688 bytes 2024-05-17T14:56:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 491748352 bytes 2024-05-17T14:56:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 532176896 bytes 2024-05-17T14:56:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 606669824 bytes 2024-05-17T14:56:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 645676032 bytes 2024-05-17T14:56:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 690583552 bytes 2024-05-17T14:56:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 773819392 bytes 2024-05-17T14:56:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 774444032 bytes 2024-05-17T14:56:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 867814400 bytes 2024-05-17T14:56:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 921247744 bytes 2024-05-17T14:56:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 990602240 bytes 2024-05-17T14:56:08.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 1027411968 bytes 2024-05-17T14:56:08.151 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 1056795648 bytes 2024-05-17T14:56:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 1072738304 bytes 2024-05-17T14:56:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1138356224 bytes 2024-05-17T14:56:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1202144256 bytes 2024-05-17T14:56:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1232302080 bytes 2024-05-17T14:56:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1259392000 bytes 2024-05-17T14:56:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1296049152 bytes 2024-05-17T14:56:08.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1340964864 bytes 2024-05-17T14:56:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1415053312 bytes 2024-05-17T14:56:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1465204736 bytes 2024-05-17T14:56:09.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1526001664 bytes 2024-05-17T14:56:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1592264704 bytes 2024-05-17T14:56:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1637370880 bytes 2024-05-17T14:56:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1718055936 bytes 2024-05-17T14:56:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1753607168 bytes 2024-05-17T14:56:09.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1778027520 bytes 2024-05-17T14:56:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1808604160 bytes 2024-05-17T14:56:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1812632576 bytes 2024-05-17T14:56:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1824487424 bytes 2024-05-17T14:56:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1851666432 bytes 2024-05-17T14:56:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1876702208 bytes 2024-05-17T14:56:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1877341184 bytes 2024-05-17T14:56:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1920124928 bytes 2024-05-17T14:56:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1920714752 bytes 2024-05-17T14:56:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1921882112 bytes 2024-05-17T14:56:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1969214464 bytes 2024-05-17T14:56:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2055757824 bytes 2024-05-17T14:56:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2081550336 bytes 2024-05-17T14:56:10.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2148246528 bytes 2024-05-17T14:56:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2185424896 bytes 2024-05-17T14:56:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2224634880 bytes 2024-05-17T14:56:11.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2244644864 bytes 2024-05-17T14:56:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2334974976 bytes 2024-05-17T14:56:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2342310912 bytes 2024-05-17T14:56:11.342 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2344039424 bytes 2024-05-17T14:56:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2439365632 bytes 2024-05-17T14:56:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2464365568 bytes 2024-05-17T14:56:11.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2515023872 bytes 2024-05-17T14:56:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2526896128 bytes 2024-05-17T14:56:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2529041408 bytes 2024-05-17T14:56:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2608508928 bytes 2024-05-17T14:56:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2634946560 bytes 2024-05-17T14:56:12.120 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2662681600 bytes 2024-05-17T14:56:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2709304320 bytes 2024-05-17T14:56:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2760041472 bytes 2024-05-17T14:56:12.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2840086528 bytes 2024-05-17T14:56:12.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2885868544 bytes 2024-05-17T14:56:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2886472704 bytes 2024-05-17T14:56:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2932363264 bytes 2024-05-17T14:56:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2960314368 bytes 2024-05-17T14:56:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 3048574976 bytes 2024-05-17T14:56:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3052783616 bytes 2024-05-17T14:56:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3120336896 bytes 2024-05-17T14:56:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3136169984 bytes 2024-05-17T14:56:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3200527360 bytes 2024-05-17T14:56:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3261063168 bytes 2024-05-17T14:56:13.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3323967488 bytes 2024-05-17T14:56:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3357097984 bytes 2024-05-17T14:56:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3448247296 bytes 2024-05-17T14:56:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3466919936 bytes 2024-05-17T14:56:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3517606912 bytes 2024-05-17T14:56:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3546925056 bytes 2024-05-17T14:56:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3559968768 bytes 2024-05-17T14:56:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3618868224 bytes 2024-05-17T14:56:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3646539776 bytes 2024-05-17T14:56:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3667768320 bytes 2024-05-17T14:56:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3679248384 bytes 2024-05-17T14:56:14.691 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3720994816 bytes 2024-05-17T14:56:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3764352000 bytes 2024-05-17T14:56:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3822848000 bytes 2024-05-17T14:56:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3913564160 bytes 2024-05-17T14:56:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 3981976576 bytes 2024-05-17T14:56:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9200 IOs, 3982578688 bytes 2024-05-17T14:56:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9300 IOs, 4007267328 bytes 2024-05-17T14:56:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9400 IOs, 4100941824 bytes 2024-05-17T14:56:15.694 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9500 IOs, 4188206080 bytes 2024-05-17T14:56:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9600 IOs, 4207056896 bytes 2024-05-17T14:56:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9700 IOs, 4224589824 bytes 2024-05-17T14:56:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9800 IOs, 4238493696 bytes 2024-05-17T14:56:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9900 IOs, 4265458688 bytes 2024-05-17T14:56:16.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10000 IOs, 4301702144 bytes 2024-05-17T14:56:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10100 IOs, 4363278336 bytes 2024-05-17T14:56:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10200 IOs, 4392477696 bytes 2024-05-17T14:56:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10300 IOs, 4539042816 bytes 2024-05-17T14:56:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10400 IOs, 4595012608 bytes 2024-05-17T14:56:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T14:56:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T14:56:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T14:56:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:56:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-17T14:56:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:56:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:56:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:56:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:56:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T14:56:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T14:56:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:56:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:56:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-17T14:56:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:56:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:56:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:56:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:56:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T14:56:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T14:56:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T14:56:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T14:56:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-17T14:56:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='test: 2024-05-17T14:56:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: global_id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T14:56:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+replaying 2024-05-17T14:56:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: replaying, {"bytes_per_second":17841561.6,"bytes_per_snapshot":535565824.0,"last_snapshot_bytes":535246848,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957756,"remote_snapshot_timestamp":1715957756,"replay_state":"idle"} 2024-05-17T14:56:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: service: mirror.0 on smithi195 2024-05-17T14:56:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 14:56:16 2024-05-17T14:56:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: peer_sites: 2024-05-17T14:56:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: name: cluster2 2024-05-17T14:56:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+stopped 2024-05-17T14:56:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: local image is primary 2024-05-17T14:56:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 14:56:16' 2024-05-17T14:56:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ state: .*up+replaying' 2024-05-17T14:56:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T14:56:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ description: .*' 2024-05-17T14:56:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":17841561.6,"bytes_per_snapshot":535565824.0,"last_snapshot_bytes":535246848,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957756,"remote_snapshot_timestamp":1715957756,"replay_state":"idle"} 2024-05-17T14:56:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:56:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T14:56:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^up+' 2024-05-17T14:56:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T14:56:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep 'service: *mirror..* on ' 2024-05-17T14:56:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.0 on smithi195 2024-05-17T14:56:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update 2024-05-17T14:56:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-17T14:56:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 14:56:16' 2024-05-17T14:56:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-17 14:56:16' 2024-05-17T14:56:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:56:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:56:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:56:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:56:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local 'prev_last_update=2024-05-17 14:56:16' 2024-05-17T14:56:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T14:56:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T14:56:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-17T14:56:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='OKOKOK142114285mirror.0smithi19519.0.0-3735-gee59d86atrueOK42144288mirror.1smithi19519.0.0-3735-gee59d86afalseOK42184296mirror.2smithi19519.0.0-3735-gee59d86afalseOK42224307mirror.3smithi19519.0.0-3735-gee59d86afalseOKtest72894289-d917-4c92-922e-8f6ade3b96caup+replayingreplaying, {"bytes_per_second":17841561.6,"bytes_per_snapshot":535565824.0,"last_snapshot_bytes":535246848,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957756,"remote_snapshot_timestamp":1715957756,"replay_state":"idle"}42114285mirror.0smithi1952024-05-17 14:56:16cluster27e816122-c11e-4a54-9bab-7372f406213bup+stoppedlocal image is primary2024-05-17 14:56:16' 2024-05-17T14:56:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update state 2024-05-17T14:56:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-17T14:56:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 14:56:16' 2024-05-17T14:56:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-17T14:56:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ state=up+replaying 2024-05-17T14:56:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T14:56:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep up+replaying 2024-05-17T14:56:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T14:56:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:56:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:56:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap3 2024-05-17T14:56:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap3 2024-05-17T14:56:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:56:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:56:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:56:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap3 2024-05-17T14:56:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-05-17T14:56:30.725 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-05-17T14:56:30.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-05-17T14:56:30.726 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-05-17T14:56:30.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T14:56:30.726+0000 7f297619d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 37974) UID: 1000 2024-05-17T14:56:35.726 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-05-17T14:56:35.726 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T14:56:35.727 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-05-17T14:56:35.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-05-17T14:56:35.727 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-05-17T14:56:35.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-05-17T14:56:35.731 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2024-05-17T14:56:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-17T14:56:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:56:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:56:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:56:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-17T14:56:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:56:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:56:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:56:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T14:56:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T14:56:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:56:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:56:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-17T14:56:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:56:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:56:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:56:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T14:56:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T14:56:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T14:56:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-17T14:56:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T14:56:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:56:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T14:56:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T14:56:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T14:56:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T14:56:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:56:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T14:56:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T14:56:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T14:56:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:56:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T14:56:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T14:56:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:56:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:56:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:56:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T14:56:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:56:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:56:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:56:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:56:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T14:56:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:56:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T14:56:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:56:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T14:56:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T14:56:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:56:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:56:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T14:56:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T14:56:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:56:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:56:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:56:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T14:56:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:56:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:56:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T14:56:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T14:56:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T14:56:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:56:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:56:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:56:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T14:56:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:56:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:56:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:56:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:56:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T14:56:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T14:56:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:56:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:56:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T14:56:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T14:56:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:56:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:56:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:56:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T14:56:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T14:56:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:56:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T14:56:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:56:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T14:56:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T14:56:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:56:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:56:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T14:56:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T14:56:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:56:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:56:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:56:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T14:56:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:56:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:56:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T14:56:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T14:56:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T14:56:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:56:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:56:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:56:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T14:56:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:56:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:56:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:56:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:56:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T14:56:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T14:56:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:56:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:56:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T14:56:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T14:56:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:56:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:56:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:56:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T14:56:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T14:56:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:56:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T14:56:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:56:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T14:56:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T14:56:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:56:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:56:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T14:56:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T14:56:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:56:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:56:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:56:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T14:56:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:56:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:56:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T14:56:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T14:56:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T14:56:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:56:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:56:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:56:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T14:56:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:56:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:56:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:56:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:56:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-05-17T14:56:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T14:56:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:56:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:56:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T14:56:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T14:56:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:56:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:56:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:56:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T14:56:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T14:56:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:56:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T14:56:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:56:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T14:56:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T14:56:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:56:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:56:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T14:56:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T14:56:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:56:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:56:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:56:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T14:56:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:56:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:56:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T14:56:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T14:56:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T14:56:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:56:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:56:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:56:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-05-17T14:56:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:56:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:56:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:56:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:56:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 1 2024-05-17T14:56:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T14:56:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T14:56:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:56:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T14:56:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T14:56:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:56:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:56:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:56:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T14:56:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:56:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:56:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:56:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:56:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T14:56:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:56:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T14:56:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:56:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T14:56:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T14:56:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:56:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:56:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T14:56:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T14:56:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:56:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:56:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:56:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T14:56:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:56:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:56:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T14:56:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T14:56:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T14:56:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:56:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:56:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:56:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T14:56:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:56:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:56:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:56:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:56:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T14:56:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T14:56:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:56:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:56:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T14:56:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T14:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T14:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T14:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T14:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T14:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T14:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:56:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T14:56:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T14:56:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:56:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:56:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:56:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T14:56:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:56:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:56:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T14:56:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T14:56:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T14:56:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:56:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:56:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:56:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T14:56:49.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:56:49.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:56:49.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:56:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:56:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T14:56:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T14:56:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:56:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:56:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T14:56:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T14:56:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:56:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:56:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:56:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T14:56:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T14:56:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:56:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T14:56:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:56:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T14:56:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T14:56:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:56:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:56:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T14:56:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T14:56:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:56:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:56:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:56:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T14:56:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:56:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:56:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T14:56:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T14:56:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T14:56:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:56:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:56:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:56:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T14:56:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:56:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:56:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:56:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:56:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-05-17T14:56:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T14:56:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:56:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:56:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T14:56:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T14:56:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:56:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:56:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:56:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T14:56:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T14:56:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:56:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T14:56:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:56:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T14:56:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T14:56:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:56:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:56:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T14:56:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T14:56:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:56:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:56:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:56:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T14:56:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:56:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:56:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T14:56:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T14:56:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T14:56:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:56:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:56:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:56:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-05-17T14:56:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:56:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:56:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:56:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:56:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 2 2024-05-17T14:56:52.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T14:56:52.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T14:56:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:56:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T14:56:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T14:56:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:56:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:56:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:56:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T14:56:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:56:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:56:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:56:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:56:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T14:56:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:56:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T14:56:52.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:56:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T14:56:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T14:56:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:56:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:56:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T14:56:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T14:56:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:56:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:56:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:56:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T14:56:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:56:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:56:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T14:56:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T14:56:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T14:56:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:56:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:56:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:56:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T14:56:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:56:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:56:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:56:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:56:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T14:56:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T14:56:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:56:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:56:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T14:56:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T14:56:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:56:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:56:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:56:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T14:56:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T14:56:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:56:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T14:56:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:56:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T14:56:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T14:56:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:56:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:56:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T14:56:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T14:56:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:56:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:56:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:56:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T14:56:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:56:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:56:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T14:56:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T14:56:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T14:56:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:56:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:56:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:56:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T14:56:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:56:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:56:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:56:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:56:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T14:56:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T14:56:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:56:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:56:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T14:56:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T14:56:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:56:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:56:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:56:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T14:56:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T14:56:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:56:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T14:56:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:56:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T14:56:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T14:56:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:56:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:56:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T14:56:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T14:56:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:56:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:56:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:56:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T14:56:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:56:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:56:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T14:56:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T14:56:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T14:56:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:56:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:56:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:56:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T14:56:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:56:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:56:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:56:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:56:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-05-17T14:56:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T14:56:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:56:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:56:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T14:56:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T14:56:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:56:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:56:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:56:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T14:56:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T14:56:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:56:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T14:56:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:56:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T14:56:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T14:56:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:56:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:56:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T14:56:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T14:56:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:56:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:56:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:56:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T14:56:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:56:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:56:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T14:56:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T14:56:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T14:56:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:56:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:56:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:56:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-05-17T14:56:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:56:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:56:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:56:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:56:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 4 2024-05-17T14:56:53.834 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-05-17T14:56:53.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-05-17T14:56:53.835 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-05-17T14:56:53.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr:2024-05-17T14:56:53.835+0000 7f4741296640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 38088) UID: 1000 2024-05-17T14:56:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T14:56:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T14:56:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:56:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T14:56:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T14:56:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:56:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:56:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:56:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T14:56:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:56:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:56:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:56:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:56:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T14:56:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:56:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T14:56:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:56:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T14:56:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T14:56:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:56:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:56:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T14:56:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T14:56:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:56:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:56:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:56:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T14:56:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:56:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:56:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T14:56:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T14:56:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T14:56:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:56:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:56:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:56:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T14:56:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:56:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:56:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:56:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:56:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T14:56:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T14:56:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:56:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:56:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T14:56:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T14:56:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:56:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:56:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:56:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T14:56:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T14:56:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:56:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T14:56:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:56:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T14:56:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T14:56:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:56:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:56:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T14:56:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T14:56:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:56:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:56:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:56:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T14:56:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:56:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:56:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T14:56:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T14:56:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T14:56:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:56:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:56:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:56:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T14:56:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:56:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:56:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:56:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:56:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T14:56:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T14:56:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:56:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:56:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T14:56:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T14:56:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:56:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:56:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:56:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T14:56:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T14:56:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:56:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T14:56:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:56:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T14:56:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T14:56:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:56:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:56:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T14:56:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T14:56:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:56:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:56:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T14:56:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T14:56:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:56:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:56:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T14:56:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T14:56:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T14:56:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:56:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:56:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T14:56:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T14:56:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:56:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:56:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:56:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:56:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-05-17T14:56:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T14:56:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:56:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:56:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T14:56:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T14:56:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:56:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:56:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:56:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T14:56:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T14:56:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:56:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T14:56:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:56:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T14:56:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T14:56:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:56:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:56:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T14:56:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T14:56:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:56:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:56:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:56:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T14:56:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T14:56:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:56:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:56:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T14:56:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T14:56:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T14:56:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:56:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:56:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T14:56:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-05-17T14:56:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:56:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:56:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:56:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:56:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 8 2024-05-17T14:56:58.836 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-05-17T14:56:58.836 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T14:56:58.837 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-05-17T14:56:58.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-05-17T14:56:58.837 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-05-17T14:56:58.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-05-17T14:56:58.840 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2024-05-17T14:57:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T14:57:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T14:57:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:57:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T14:57:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T14:57:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:57:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:57:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:57:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T14:57:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:57:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:57:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:57:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:57:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T14:57:04.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:57:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:57:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T14:57:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:57:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T14:57:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T14:57:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:57:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:57:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T14:57:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T14:57:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:57:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:57:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:57:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:57:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T14:57:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:57:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:57:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T14:57:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T14:57:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T14:57:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:57:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:57:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:57:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T14:57:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:57:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:57:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:57:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:57:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T14:57:05.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T14:57:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:57:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:57:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T14:57:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T14:57:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:57:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:57:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:57:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T14:57:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T14:57:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:57:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:57:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T14:57:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:57:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T14:57:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T14:57:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:57:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:57:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T14:57:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T14:57:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:57:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:57:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:57:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:57:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T14:57:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:57:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:57:05.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T14:57:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T14:57:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T14:57:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:57:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:57:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:57:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T14:57:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T14:57:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T14:57:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T14:57:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T14:57:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T14:57:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T14:57:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T14:57:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:57:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-17T14:57:05.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T14:57:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:57:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T14:57:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T14:57:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-05-17T14:57:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T14:57:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:57:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-test.status 2024-05-17T14:57:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-test.status 2024-05-17T14:57:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T14:57:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:57:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-test.status 2024-05-17T14:57:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-test.status 2024-05-17T14:57:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T14:57:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:57:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-05-17T14:57:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:57:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:05.114 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-05-17T14:57:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 11 2024-05-17T14:57:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:57:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:57:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:57:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-05-17T14:57:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:57:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T14:57:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=11 2024-05-17T14:57:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:57:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:57:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T14:57:06.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:57:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:57:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:57:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:57:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:57:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:57:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:57:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T14:57:06.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:57:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T14:57:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T14:57:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 11 = '' 2024-05-17T14:57:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:57:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T14:57:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:57:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:57:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:57:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:57:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:57:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:57:07.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:57:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T14:57:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:57:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T14:57:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T14:57:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 11 = '' 2024-05-17T14:57:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:57:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T14:57:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:57:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:57:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:57:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:57:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:57:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:57:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:57:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T14:57:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:57:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T14:57:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T14:57:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 11 = '' 2024-05-17T14:57:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:57:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T14:57:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:57:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:57:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:57:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:57:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:57:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:57:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=11 2024-05-17T14:57:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 11 = 11 2024-05-17T14:57:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:57:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-05-17T14:57:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:57:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap3 2024-05-17T14:57:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T14:57:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T14:57:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:57:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-05-17T14:57:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'test': 2024-05-17T14:57:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 512 MiB in 128 objects 2024-05-17T14:57:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T14:57:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 6 2024-05-17T14:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10bd697942f 2024-05-17T14:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10bd697942f 2024-05-17T14:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T14:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T14:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T14:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T14:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 14:54:18 2024 2024-05-17T14:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 14:56:02 2024 2024-05-17T14:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 14:54:18 2024 2024-05-17T14:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T14:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T14:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T14:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T14:57:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T14:57:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:57:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snap2-clean ']' 2024-05-17T14:57:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap2-clean 2024-05-17T14:57:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap2-clean 2024-05-17T14:57:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T14:57:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:57:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:57:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:57:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:57:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:57:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:57:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:57:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:57:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:57:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:57:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap3 2024-05-17T14:57:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap3 2024-05-17T14:57:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T14:57:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:57:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:57:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:57:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:57:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:57:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:57:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:57:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:57:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:57:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:57:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ clean_snap_name=snap3-clean 2024-05-17T14:57:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-05-17T14:57:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:57:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap3-clean 2024-05-17T14:57:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-05-17T14:57:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T14:57:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror test 2024-05-17T14:57:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:57:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T14:57:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=13 2024-05-17T14:57:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T14:57:14.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-17T14:57:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 29763584 bytes 2024-05-17T14:57:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 67417088 bytes 2024-05-17T14:57:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 138398720 bytes 2024-05-17T14:57:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 196061184 bytes 2024-05-17T14:57:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 224048128 bytes 2024-05-17T14:57:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 276739072 bytes 2024-05-17T14:57:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 358026240 bytes 2024-05-17T14:57:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 403263488 bytes 2024-05-17T14:57:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 448387072 bytes 2024-05-17T14:57:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 509087744 bytes 2024-05-17T14:57:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 557604864 bytes 2024-05-17T14:57:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 607770624 bytes 2024-05-17T14:57:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 639850496 bytes 2024-05-17T14:57:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 660630528 bytes 2024-05-17T14:57:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 686120960 bytes 2024-05-17T14:57:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 701020160 bytes 2024-05-17T14:57:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 787014656 bytes 2024-05-17T14:57:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 787630080 bytes 2024-05-17T14:57:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 900756480 bytes 2024-05-17T14:57:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 914237440 bytes 2024-05-17T14:57:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 931119104 bytes 2024-05-17T14:57:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 954727424 bytes 2024-05-17T14:57:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1032498176 bytes 2024-05-17T14:57:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1041123328 bytes 2024-05-17T14:57:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1066235904 bytes 2024-05-17T14:57:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1066875904 bytes 2024-05-17T14:57:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1146256384 bytes 2024-05-17T14:57:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1151719424 bytes 2024-05-17T14:57:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1170658304 bytes 2024-05-17T14:57:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1187144704 bytes 2024-05-17T14:57:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1207673856 bytes 2024-05-17T14:57:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1291297792 bytes 2024-05-17T14:57:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1308749824 bytes 2024-05-17T14:57:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1309340672 bytes 2024-05-17T14:57:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1333925888 bytes 2024-05-17T14:57:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1449041920 bytes 2024-05-17T14:57:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1483706368 bytes 2024-05-17T14:57:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1534173184 bytes 2024-05-17T14:57:19.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1593230336 bytes 2024-05-17T14:57:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1651352576 bytes 2024-05-17T14:57:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1695906816 bytes 2024-05-17T14:57:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1781295104 bytes 2024-05-17T14:57:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1834229760 bytes 2024-05-17T14:57:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1854400512 bytes 2024-05-17T14:57:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1888947200 bytes 2024-05-17T14:57:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1889546240 bytes 2024-05-17T14:57:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1964655616 bytes 2024-05-17T14:57:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 1982029824 bytes 2024-05-17T14:57:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2039789568 bytes 2024-05-17T14:57:20.159 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2056397824 bytes 2024-05-17T14:57:20.276 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2100203520 bytes 2024-05-17T14:57:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2135741440 bytes 2024-05-17T14:57:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2203450368 bytes 2024-05-17T14:57:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2204064768 bytes 2024-05-17T14:57:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2259972096 bytes 2024-05-17T14:57:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2284914688 bytes 2024-05-17T14:57:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2320484352 bytes 2024-05-17T14:57:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2335570944 bytes 2024-05-17T14:57:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2351056896 bytes 2024-05-17T14:57:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2422376448 bytes 2024-05-17T14:57:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2470146048 bytes 2024-05-17T14:57:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2484523008 bytes 2024-05-17T14:57:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2552159232 bytes 2024-05-17T14:57:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2563241984 bytes 2024-05-17T14:57:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2594905088 bytes 2024-05-17T14:57:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2622613504 bytes 2024-05-17T14:57:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2656194560 bytes 2024-05-17T14:57:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2656800768 bytes 2024-05-17T14:57:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2681818112 bytes 2024-05-17T14:57:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 2697729024 bytes 2024-05-17T14:57:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 2747057152 bytes 2024-05-17T14:57:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 2789981184 bytes 2024-05-17T14:57:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 2801878016 bytes 2024-05-17T14:57:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 2802512896 bytes 2024-05-17T14:57:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 2848939008 bytes 2024-05-17T14:57:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 2855497728 bytes 2024-05-17T14:57:22.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 2928037888 bytes 2024-05-17T14:57:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 2987977728 bytes 2024-05-17T14:57:22.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3003338752 bytes 2024-05-17T14:57:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3030070272 bytes 2024-05-17T14:57:23.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3072092160 bytes 2024-05-17T14:57:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3129700352 bytes 2024-05-17T14:57:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3168530432 bytes 2024-05-17T14:57:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3206047744 bytes 2024-05-17T14:57:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3262299136 bytes 2024-05-17T14:57:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3298144256 bytes 2024-05-17T14:57:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3332431872 bytes 2024-05-17T14:57:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3351150592 bytes 2024-05-17T14:57:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3394778112 bytes 2024-05-17T14:57:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3437488128 bytes 2024-05-17T14:57:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 3465446400 bytes 2024-05-17T14:57:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9200 IOs, 3534867456 bytes 2024-05-17T14:57:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9300 IOs, 3573648384 bytes 2024-05-17T14:57:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9400 IOs, 3635545088 bytes 2024-05-17T14:57:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9500 IOs, 3663021056 bytes 2024-05-17T14:57:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9600 IOs, 3718474752 bytes 2024-05-17T14:57:24.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9700 IOs, 3752734720 bytes 2024-05-17T14:57:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9800 IOs, 3776911360 bytes 2024-05-17T14:57:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9900 IOs, 3798700032 bytes 2024-05-17T14:57:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10000 IOs, 3848904704 bytes 2024-05-17T14:57:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10100 IOs, 3904060416 bytes 2024-05-17T14:57:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10200 IOs, 3934321664 bytes 2024-05-17T14:57:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10300 IOs, 3959897088 bytes 2024-05-17T14:57:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10400 IOs, 3994345472 bytes 2024-05-17T14:57:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10500 IOs, 4000032768 bytes 2024-05-17T14:57:25.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10600 IOs, 4031501312 bytes 2024-05-17T14:57:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10700 IOs, 4125384704 bytes 2024-05-17T14:57:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10800 IOs, 4155125760 bytes 2024-05-17T14:57:25.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10900 IOs, 4167630848 bytes 2024-05-17T14:57:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11000 IOs, 4198394880 bytes 2024-05-17T14:57:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11100 IOs, 4272991232 bytes 2024-05-17T14:57:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11200 IOs, 4316723200 bytes 2024-05-17T14:57:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11300 IOs, 4369226752 bytes 2024-05-17T14:57:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11400 IOs, 4434728960 bytes 2024-05-17T14:57:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11500 IOs, 4486791168 bytes 2024-05-17T14:57:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11600 IOs, 4535600128 bytes 2024-05-17T14:57:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11700 IOs, 4587404288 bytes 2024-05-17T14:57:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11800 IOs, 4606227456 bytes 2024-05-17T14:57:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11900 IOs, 4656075776 bytes 2024-05-17T14:57:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:12000 IOs, 4681184256 bytes 2024-05-17T14:57:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:12100 IOs, 4681803776 bytes 2024-05-17T14:57:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:12200 IOs, 4715383808 bytes 2024-05-17T14:57:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:12300 IOs, 4733995008 bytes 2024-05-17T14:57:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:12400 IOs, 4758266880 bytes 2024-05-17T14:57:27.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:12500 IOs, 4761892864 bytes 2024-05-17T14:57:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:12600 IOs, 4786902016 bytes 2024-05-17T14:57:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:12700 IOs, 4807397376 bytes 2024-05-17T14:57:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T14:57:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T14:57:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T14:57:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:57:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-17T14:57:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:57:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:57:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T14:57:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T14:57:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:57:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:57:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-17T14:57:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:57:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:57:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T14:57:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T14:57:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T14:57:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T14:57:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-17T14:57:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='test: 2024-05-17T14:57:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: global_id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T14:57:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+replaying 2024-05-17T14:57:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: replaying, {"bytes_per_second":48452049.45,"bytes_per_snapshot":532972544.0,"last_snapshot_bytes":532972544,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957825,"remote_snapshot_timestamp":1715957825,"replay_state":"idle"} 2024-05-17T14:57:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: service: mirror.1 on smithi195 2024-05-17T14:57:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 14:57:16 2024-05-17T14:57:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: peer_sites: 2024-05-17T14:57:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: name: cluster2 2024-05-17T14:57:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+stopped 2024-05-17T14:57:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: local image is primary 2024-05-17T14:57:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 14:57:27' 2024-05-17T14:57:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ state: .*up+replaying' 2024-05-17T14:57:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T14:57:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ description: .*' 2024-05-17T14:57:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":48452049.45,"bytes_per_snapshot":532972544.0,"last_snapshot_bytes":532972544,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957825,"remote_snapshot_timestamp":1715957825,"replay_state":"idle"} 2024-05-17T14:57:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:57:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T14:57:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^up+' 2024-05-17T14:57:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T14:57:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep 'service: *mirror..* on ' 2024-05-17T14:57:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.1 on smithi195 2024-05-17T14:57:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update 2024-05-17T14:57:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-17T14:57:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 14:57:16' 2024-05-17T14:57:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-17 14:57:16' 2024-05-17T14:57:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:57:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:57:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local 'prev_last_update=2024-05-17 14:57:16' 2024-05-17T14:57:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T14:57:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T14:57:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-17T14:57:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='OKOKOK142144288mirror.1smithi19519.0.0-3735-gee59d86atrueOK42184296mirror.2smithi19519.0.0-3735-gee59d86afalseOK42224307mirror.3smithi19519.0.0-3735-gee59d86afalseOK44024405mirror.0smithi19519.0.0-3735-gee59d86afalseOKtest72894289-d917-4c92-922e-8f6ade3b96caup+replayingreplaying, {"bytes_per_second":48452049.45,"bytes_per_snapshot":532972544.0,"last_snapshot_bytes":532972544,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957825,"remote_snapshot_timestamp":1715957825,"replay_state":"idle"}42144288mirror.1smithi1952024-05-17 14:57:16cluster27e816122-c11e-4a54-9bab-7372f406213bup+stoppedlocal image is primary2024-05-17 14:57:27' 2024-05-17T14:57:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update state 2024-05-17T14:57:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-17T14:57:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 14:57:16' 2024-05-17T14:57:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-17T14:57:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ state=up+replaying 2024-05-17T14:57:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T14:57:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep up+replaying 2024-05-17T14:57:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T14:57:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:57:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:57:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap4 2024-05-17T14:57:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap4 2024-05-17T14:57:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:57:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap4 2024-05-17T14:57:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-05-17T14:57:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-17T14:57:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:57:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-17T14:57:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:57:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T14:57:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T14:57:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:57:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:57:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-17T14:57:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:57:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T14:57:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T14:57:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T14:57:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-17T14:57:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T14:57:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T14:57:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:57:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T14:57:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T14:57:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T14:57:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:57:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T14:57:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T14:57:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T14:57:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:57:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T14:57:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T14:57:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:57:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:57:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:57:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T14:57:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:57:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:57:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:57:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:57:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T14:57:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:57:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:57:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T14:57:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:57:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T14:57:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T14:57:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:57:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:57:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T14:57:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T14:57:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:57:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:57:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:57:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:57:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T14:57:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:57:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:57:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T14:57:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T14:57:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T14:57:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:57:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:57:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:57:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T14:57:57.431 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:57:57.432 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:57:57.432 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:57:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:57:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T14:57:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T14:57:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:57:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:57:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T14:57:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T14:57:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:57:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:57:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:57:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T14:57:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T14:57:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:57:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:57:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T14:57:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:57:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T14:57:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T14:57:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:57:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T14:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T14:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T14:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T14:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T14:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T14:57:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:57:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:57:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:57:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T14:57:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T14:57:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T14:57:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T14:57:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T14:57:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T14:57:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T14:57:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T14:57:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:57:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-17T14:57:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T14:57:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:57:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T14:57:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T14:57:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-05-17T14:57:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T14:57:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:57:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-test.status 2024-05-17T14:57:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-test.status 2024-05-17T14:57:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T14:57:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:57:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-test.status 2024-05-17T14:57:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-test.status 2024-05-17T14:57:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T14:57:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:57:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-05-17T14:57:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:57:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-05-17T14:57:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 14 2024-05-17T14:57:59.342 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:57:59.342 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:57:59.342 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:57:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:57:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-05-17T14:57:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T14:57:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=14 2024-05-17T14:57:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:57:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:57:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T14:57:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:57:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:57:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:57:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:57:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:57:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:57:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:57:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:57:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:57:59.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T14:57:59.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:57:59.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T14:57:59.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T14:57:59.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 14 = '' 2024-05-17T14:57:59.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:57:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T14:58:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:58:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:58:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:58:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:58:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:58:00.117 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:58:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:58:00.119 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T14:58:00.119 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:58:00.119 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T14:58:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T14:58:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 14 = '' 2024-05-17T14:58:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:58:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T14:58:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:58:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:58:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:58:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:58:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:58:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:58:00.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:58:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T14:58:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:58:00.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T14:58:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T14:58:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 14 = '' 2024-05-17T14:58:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:58:00.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T14:58:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:58:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:58:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:58:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:58:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:58:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:58:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=14 2024-05-17T14:58:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 14 = 14 2024-05-17T14:58:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:58:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-05-17T14:58:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:58:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap4 2024-05-17T14:58:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T14:58:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T14:58:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:58:02.732 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6609285994456775) > weight (0.25) 2024-05-17T14:58:02.732 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3546900394295327) > weight (0.25) 2024-05-17T14:58:02.732 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5843142735566356) > weight (0.25) 2024-05-17T14:58:02.732 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9184081357960914) > weight (0.25) 2024-05-17T14:58:02.732 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2024-05-17T14:58:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-05-17T14:58:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'test': 2024-05-17T14:58:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 512 MiB in 128 objects 2024-05-17T14:58:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T14:58:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 8 2024-05-17T14:58:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10bd697942f 2024-05-17T14:58:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10bd697942f 2024-05-17T14:58:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T14:58:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T14:58:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T14:58:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T14:58:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 14:54:18 2024 2024-05-17T14:58:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 14:57:11 2024 2024-05-17T14:58:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 14:54:18 2024 2024-05-17T14:58:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T14:58:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T14:58:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T14:58:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T14:58:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T14:58:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:58:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snap3-clean ']' 2024-05-17T14:58:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap3-clean 2024-05-17T14:58:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap3-clean 2024-05-17T14:58:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T14:58:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:58:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:58:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:58:04.308 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:58:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:58:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap4 2024-05-17T14:58:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap4 2024-05-17T14:58:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T14:58:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:58:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:58:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:58:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:58:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:58:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ clean_snap_name=snap4-clean 2024-05-17T14:58:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-05-17T14:58:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:58:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:07.164 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap4-clean 2024-05-17T14:58:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-05-17T14:58:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T14:58:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror test 2024-05-17T14:58:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:58:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T14:58:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=6 2024-05-17T14:58:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T14:58:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-17T14:58:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 21933056 bytes 2024-05-17T14:58:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 57608192 bytes 2024-05-17T14:58:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 91813888 bytes 2024-05-17T14:58:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 189880320 bytes 2024-05-17T14:58:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 231709696 bytes 2024-05-17T14:58:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 277650432 bytes 2024-05-17T14:58:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 330761216 bytes 2024-05-17T14:58:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 356245504 bytes 2024-05-17T14:58:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 395506688 bytes 2024-05-17T14:58:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 444778496 bytes 2024-05-17T14:58:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 472856576 bytes 2024-05-17T14:58:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 506161152 bytes 2024-05-17T14:58:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 539327488 bytes 2024-05-17T14:58:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 539954176 bytes 2024-05-17T14:58:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 568511488 bytes 2024-05-17T14:58:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 623070208 bytes 2024-05-17T14:58:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 665076736 bytes 2024-05-17T14:58:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 786042880 bytes 2024-05-17T14:58:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 898404352 bytes 2024-05-17T14:58:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 927500288 bytes 2024-05-17T14:58:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 974729216 bytes 2024-05-17T14:58:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1045359616 bytes 2024-05-17T14:58:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1078431744 bytes 2024-05-17T14:58:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1156431872 bytes 2024-05-17T14:58:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1219375104 bytes 2024-05-17T14:58:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1228502016 bytes 2024-05-17T14:58:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1266523136 bytes 2024-05-17T14:58:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1271320576 bytes 2024-05-17T14:58:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1349778432 bytes 2024-05-17T14:58:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1350396928 bytes 2024-05-17T14:58:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1373498368 bytes 2024-05-17T14:58:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1378506752 bytes 2024-05-17T14:58:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1416988672 bytes 2024-05-17T14:58:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1434193920 bytes 2024-05-17T14:58:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1493344256 bytes 2024-05-17T14:58:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1575880704 bytes 2024-05-17T14:58:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1633201152 bytes 2024-05-17T14:58:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1669749760 bytes 2024-05-17T14:58:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1675832320 bytes 2024-05-17T14:58:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1723502592 bytes 2024-05-17T14:58:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1782515712 bytes 2024-05-17T14:58:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1783116800 bytes 2024-05-17T14:58:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1831469056 bytes 2024-05-17T14:58:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1906703360 bytes 2024-05-17T14:58:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1935637504 bytes 2024-05-17T14:58:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1986194432 bytes 2024-05-17T14:58:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2031906816 bytes 2024-05-17T14:58:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2047945728 bytes 2024-05-17T14:58:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2076152832 bytes 2024-05-17T14:58:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2098091008 bytes 2024-05-17T14:58:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2157687808 bytes 2024-05-17T14:58:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2196549632 bytes 2024-05-17T14:58:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2197182464 bytes 2024-05-17T14:58:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2235596800 bytes 2024-05-17T14:58:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T14:58:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T14:58:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T14:58:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:58:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-17T14:58:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:58:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:58:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T14:58:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T14:58:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:58:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:58:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-17T14:58:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:58:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:58:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T14:58:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T14:58:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T14:58:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T14:58:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-17T14:58:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='test: 2024-05-17T14:58:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: global_id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T14:58:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+replaying 2024-05-17T14:58:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":532972544.0,"last_snapshot_bytes":532972544,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957825,"remote_snapshot_timestamp":1715957825,"replay_state":"idle"} 2024-05-17T14:58:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: service: mirror.1 on smithi195 2024-05-17T14:58:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 14:57:46 2024-05-17T14:58:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: peer_sites: 2024-05-17T14:58:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: name: cluster2 2024-05-17T14:58:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+stopped 2024-05-17T14:58:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: local image is primary 2024-05-17T14:58:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 14:57:46' 2024-05-17T14:58:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ state: .*up+replaying' 2024-05-17T14:58:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T14:58:14.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ description: .*' 2024-05-17T14:58:14.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":532972544.0,"last_snapshot_bytes":532972544,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957825,"remote_snapshot_timestamp":1715957825,"replay_state":"idle"} 2024-05-17T14:58:14.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:58:14.417 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T14:58:14.417 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^up+' 2024-05-17T14:58:14.417 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T14:58:14.417 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep 'service: *mirror..* on ' 2024-05-17T14:58:14.418 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.1 on smithi195 2024-05-17T14:58:14.418 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update 2024-05-17T14:58:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-17T14:58:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 14:57:46' 2024-05-17T14:58:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-17 14:57:46' 2024-05-17T14:58:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:58:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:58:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local 'prev_last_update=2024-05-17 14:57:46' 2024-05-17T14:58:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T14:58:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T14:58:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-17T14:58:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='OKOKOK142144288mirror.1smithi19519.0.0-3735-gee59d86atrueOK42184296mirror.2smithi19519.0.0-3735-gee59d86afalseOK42224307mirror.3smithi19519.0.0-3735-gee59d86afalseOK44024405mirror.0smithi19519.0.0-3735-gee59d86afalseOKtest72894289-d917-4c92-922e-8f6ade3b96caup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":532972544.0,"last_snapshot_bytes":532972544,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957825,"remote_snapshot_timestamp":1715957825,"replay_state":"idle"}42144288mirror.1smithi1952024-05-17 14:57:46cluster27e816122-c11e-4a54-9bab-7372f406213bup+stoppedlocal image is primary2024-05-17 14:57:46' 2024-05-17T14:58:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update state 2024-05-17T14:58:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-17T14:58:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 14:57:46' 2024-05-17T14:58:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-17T14:58:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ state=up+replaying 2024-05-17T14:58:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T14:58:14.480 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep up+replaying 2024-05-17T14:58:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T14:58:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:58:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:58:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap5 2024-05-17T14:58:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap5 2024-05-17T14:58:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:58:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap5 2024-05-17T14:58:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-05-17T14:58:18.841 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.529153985660256) > weight (0.25) 2024-05-17T14:58:18.841 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4051123188580651) > weight (0.25) 2024-05-17T14:58:18.841 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-05-17T14:58:18.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-05-17T14:58:18.842 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-05-17T14:58:18.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi195.stderr:2024-05-17T14:58:18.843+0000 7fa793cd0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 38144) UID: 1000 2024-05-17T14:58:26.844 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-05-17T14:58:26.844 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T14:58:26.844 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-05-17T14:58:26.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-05-17T14:58:26.845 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-05-17T14:58:26.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-05-17T14:58:26.848 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2024-05-17T14:58:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-17T14:58:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:58:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-17T14:58:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:58:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T14:58:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T14:58:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:58:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:58:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-17T14:58:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:58:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T14:58:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T14:58:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T14:58:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-17T14:58:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T14:58:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:58:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T14:58:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T14:58:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T14:58:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T14:58:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:58:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T14:58:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T14:58:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T14:58:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:58:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T14:58:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T14:58:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:58:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:58:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:58:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T14:58:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:58:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:58:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:58:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:58:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T14:58:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:58:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:58:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T14:58:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:58:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T14:58:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T14:58:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:58:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:58:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T14:58:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T14:58:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:58:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:58:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:58:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:58:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T14:58:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:58:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:58:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T14:58:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T14:58:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T14:58:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:58:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:58:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:58:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T14:58:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:58:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:58:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:58:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:58:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T14:58:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T14:58:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:58:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:58:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T14:58:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T14:58:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:58:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:58:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:58:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T14:58:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T14:58:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:58:45.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:58:45.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T14:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T14:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T14:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T14:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T14:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T14:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:58:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:58:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T14:58:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T14:58:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T14:58:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:58:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:58:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:58:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T14:58:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T14:58:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T14:58:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T14:58:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T14:58:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T14:58:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T14:58:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T14:58:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:58:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-17T14:58:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T14:58:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:58:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T14:58:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T14:58:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-05-17T14:58:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T14:58:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:58:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-test.status 2024-05-17T14:58:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-test.status 2024-05-17T14:58:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T14:58:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:58:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-test.status 2024-05-17T14:58:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-test.status 2024-05-17T14:58:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T14:58:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:58:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-05-17T14:58:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:58:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-05-17T14:58:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 17 2024-05-17T14:58:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:58:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:58:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:58:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:58:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-05-17T14:58:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T14:58:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=17 2024-05-17T14:58:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:58:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:58:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T14:58:48.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:58:48.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:58:48.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:58:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:58:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:58:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:58:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:58:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T14:58:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:58:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T14:58:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T14:58:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 17 = '' 2024-05-17T14:58:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:58:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T14:58:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:58:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:58:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:58:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:58:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:58:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:58:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:58:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T14:58:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:58:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T14:58:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T14:58:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 17 = '' 2024-05-17T14:58:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:58:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T14:58:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:58:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:58:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:58:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:58:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:58:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:58:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:58:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T14:58:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:58:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T14:58:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T14:58:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 17 = '' 2024-05-17T14:58:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:58:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T14:58:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:58:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:58:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:58:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:58:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:58:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:58:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=17 2024-05-17T14:58:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 17 = 17 2024-05-17T14:58:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:58:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-05-17T14:58:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:58:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap5 2024-05-17T14:58:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T14:58:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T14:58:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:58:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-05-17T14:58:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'test': 2024-05-17T14:58:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 512 MiB in 128 objects 2024-05-17T14:58:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T14:58:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 10 2024-05-17T14:58:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10bd697942f 2024-05-17T14:58:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10bd697942f 2024-05-17T14:58:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T14:58:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T14:58:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T14:58:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T14:58:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 14:54:18 2024 2024-05-17T14:58:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 14:57:11 2024 2024-05-17T14:58:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 14:54:18 2024 2024-05-17T14:58:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T14:58:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T14:58:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T14:58:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T14:58:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T14:58:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:58:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snap4-clean ']' 2024-05-17T14:58:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap4-clean 2024-05-17T14:58:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap4-clean 2024-05-17T14:58:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T14:58:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:58:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:58:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:58:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:58:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:58:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap5 2024-05-17T14:58:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap5 2024-05-17T14:58:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T14:58:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:58:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:58:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:58:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:58:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:58:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:58:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ clean_snap_name=snap5-clean 2024-05-17T14:58:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-05-17T14:58:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:58:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap5-clean 2024-05-17T14:58:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-05-17T14:58:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T14:58:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror test 2024-05-17T14:58:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:58:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:58:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:58:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T14:58:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=7 2024-05-17T14:58:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T14:58:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-17T14:58:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 55142400 bytes 2024-05-17T14:58:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 72169472 bytes 2024-05-17T14:58:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 113425408 bytes 2024-05-17T14:58:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 156599296 bytes 2024-05-17T14:58:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 176412672 bytes 2024-05-17T14:58:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 212493312 bytes 2024-05-17T14:58:57.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 304193536 bytes 2024-05-17T14:58:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 330980352 bytes 2024-05-17T14:58:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 373854208 bytes 2024-05-17T14:58:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 374455296 bytes 2024-05-17T14:58:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 396906496 bytes 2024-05-17T14:58:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 424450048 bytes 2024-05-17T14:58:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 508341248 bytes 2024-05-17T14:58:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 533348352 bytes 2024-05-17T14:58:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 600265728 bytes 2024-05-17T14:58:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 600890368 bytes 2024-05-17T14:58:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 664975360 bytes 2024-05-17T14:58:58.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 692171776 bytes 2024-05-17T14:58:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 780778496 bytes 2024-05-17T14:58:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 787982336 bytes 2024-05-17T14:58:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 814920704 bytes 2024-05-17T14:58:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 834938880 bytes 2024-05-17T14:58:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 884048896 bytes 2024-05-17T14:58:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 964375552 bytes 2024-05-17T14:58:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 998250496 bytes 2024-05-17T14:58:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1058202624 bytes 2024-05-17T14:58:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1110743040 bytes 2024-05-17T14:58:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1111359488 bytes 2024-05-17T14:58:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1150258176 bytes 2024-05-17T14:58:59.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1173745664 bytes 2024-05-17T14:58:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1174366208 bytes 2024-05-17T14:58:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1195650048 bytes 2024-05-17T14:58:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1251665920 bytes 2024-05-17T14:59:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1291300864 bytes 2024-05-17T14:59:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1306673152 bytes 2024-05-17T14:59:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1308723200 bytes 2024-05-17T14:59:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1330416640 bytes 2024-05-17T14:59:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1392509952 bytes 2024-05-17T14:59:00.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1501733888 bytes 2024-05-17T14:59:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1552206848 bytes 2024-05-17T14:59:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1662859264 bytes 2024-05-17T14:59:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1704133632 bytes 2024-05-17T14:59:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1760881664 bytes 2024-05-17T14:59:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1825818624 bytes 2024-05-17T14:59:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1890942976 bytes 2024-05-17T14:59:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1939721216 bytes 2024-05-17T14:59:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1971255296 bytes 2024-05-17T14:59:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2020712448 bytes 2024-05-17T14:59:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2043891712 bytes 2024-05-17T14:59:01.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2044515328 bytes 2024-05-17T14:59:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2081220608 bytes 2024-05-17T14:59:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2142597120 bytes 2024-05-17T14:59:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2184026112 bytes 2024-05-17T14:59:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2212571136 bytes 2024-05-17T14:59:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2265434112 bytes 2024-05-17T14:59:02.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2276983808 bytes 2024-05-17T14:59:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2326928384 bytes 2024-05-17T14:59:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2409284608 bytes 2024-05-17T14:59:02.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2431759360 bytes 2024-05-17T14:59:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2462078976 bytes 2024-05-17T14:59:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2493378560 bytes 2024-05-17T14:59:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2570582016 bytes 2024-05-17T14:59:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T14:59:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T14:59:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T14:59:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:59:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-17T14:59:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:59:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:59:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T14:59:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T14:59:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:59:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:59:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-17T14:59:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:59:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:59:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T14:59:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T14:59:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T14:59:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T14:59:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-17T14:59:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='test: 2024-05-17T14:59:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: global_id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T14:59:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+replaying 2024-05-17T14:59:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533811200.0,"last_snapshot_bytes":534649856,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957878,"remote_snapshot_timestamp":1715957878,"replay_state":"idle"} 2024-05-17T14:59:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: service: mirror.1 on smithi195 2024-05-17T14:59:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 14:58:46 2024-05-17T14:59:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: peer_sites: 2024-05-17T14:59:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: name: cluster2 2024-05-17T14:59:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+stopped 2024-05-17T14:59:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: local image is primary 2024-05-17T14:59:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 14:58:47' 2024-05-17T14:59:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ state: .*up+replaying' 2024-05-17T14:59:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T14:59:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ description: .*' 2024-05-17T14:59:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533811200.0,"last_snapshot_bytes":534649856,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957878,"remote_snapshot_timestamp":1715957878,"replay_state":"idle"} 2024-05-17T14:59:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:59:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T14:59:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^up+' 2024-05-17T14:59:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T14:59:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep 'service: *mirror..* on ' 2024-05-17T14:59:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.1 on smithi195 2024-05-17T14:59:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update 2024-05-17T14:59:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-17T14:59:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 14:58:46' 2024-05-17T14:59:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-17 14:58:46' 2024-05-17T14:59:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:59:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:59:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local 'prev_last_update=2024-05-17 14:58:46' 2024-05-17T14:59:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T14:59:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T14:59:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-17T14:59:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='OKOKOK142144288mirror.1smithi19519.0.0-3735-gee59d86atrueOK42184296mirror.2smithi19519.0.0-3735-gee59d86afalseOK42224307mirror.3smithi19519.0.0-3735-gee59d86afalseOK44024405mirror.0smithi19519.0.0-3735-gee59d86afalseOKtest72894289-d917-4c92-922e-8f6ade3b96caup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533811200.0,"last_snapshot_bytes":534649856,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957878,"remote_snapshot_timestamp":1715957878,"replay_state":"idle"}42144288mirror.1smithi1952024-05-17 14:58:46cluster27e816122-c11e-4a54-9bab-7372f406213bup+stoppedlocal image is primary2024-05-17 14:58:47' 2024-05-17T14:59:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update state 2024-05-17T14:59:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-17T14:59:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 14:58:46' 2024-05-17T14:59:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-17T14:59:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ state=up+replaying 2024-05-17T14:59:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T14:59:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep up+replaying 2024-05-17T14:59:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T14:59:04.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:59:04.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:59:04.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap6 2024-05-17T14:59:04.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap6 2024-05-17T14:59:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:59:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap6 2024-05-17T14:59:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-05-17T14:59:06.733 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7519691996834486) > weight (0.25) 2024-05-17T14:59:06.733 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.2826193389150081) > weight (0.25) 2024-05-17T14:59:06.733 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9588929320231785) > weight (0.25) 2024-05-17T14:59:06.733 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5899378352918044) > weight (0.25) 2024-05-17T14:59:06.734 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2024-05-17T14:59:31.849 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5875481925425987) > weight (0.25) 2024-05-17T14:59:31.849 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8705662413714971) > weight (0.25) 2024-05-17T14:59:31.849 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6616857443654112) > weight (0.25) 2024-05-17T14:59:31.849 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.37584190681466856) > weight (0.25) 2024-05-17T14:59:31.850 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2024-05-17T14:59:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-17T14:59:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:59:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-17T14:59:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:59:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T14:59:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T14:59:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:59:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:59:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-17T14:59:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:59:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T14:59:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T14:59:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T14:59:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-17T14:59:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T14:59:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:59:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T14:59:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T14:59:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T14:59:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T14:59:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:59:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T14:59:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T14:59:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T14:59:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:59:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T14:59:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T14:59:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:59:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:59:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T14:59:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T14:59:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:59:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:59:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:59:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T14:59:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T14:59:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:59:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:59:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T14:59:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:59:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T14:59:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T14:59:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:59:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:59:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T14:59:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T14:59:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:59:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:59:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:59:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T14:59:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T14:59:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:59:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:59:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T14:59:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T14:59:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T14:59:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:59:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:59:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T14:59:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T14:59:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T14:59:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T14:59:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T14:59:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T14:59:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T14:59:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T14:59:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T14:59:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T14:59:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T14:59:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T14:59:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T14:59:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T14:59:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T14:59:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T14:59:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T14:59:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T14:59:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:59:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T14:59:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:59:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T14:59:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T14:59:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T14:59:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T14:59:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T14:59:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T14:59:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T14:59:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T14:59:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T14:59:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T14:59:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T14:59:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T14:59:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T14:59:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T14:59:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T14:59:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T14:59:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:59:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:59:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T14:59:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T14:59:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T14:59:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T14:59:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T14:59:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T14:59:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T14:59:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T14:59:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T14:59:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:59:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-17T14:59:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T14:59:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:59:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T14:59:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T14:59:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-05-17T14:59:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T14:59:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:59:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-test.status 2024-05-17T14:59:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-test.status 2024-05-17T14:59:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T14:59:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:59:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-test.status 2024-05-17T14:59:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-test.status 2024-05-17T14:59:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T14:59:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:59:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-05-17T14:59:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:59:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-05-17T14:59:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 20 2024-05-17T14:59:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:59:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:59:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T14:59:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-05-17T14:59:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:59:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T14:59:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=20 2024-05-17T14:59:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:59:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:59:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T14:59:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:59:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:59:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:59:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:59:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:59:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:59:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:59:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T14:59:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:59:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T14:59:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T14:59:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 20 = '' 2024-05-17T14:59:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:59:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T14:59:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:59:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:59:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:59:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:59:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:59:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:59:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:59:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T14:59:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:59:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T14:59:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T14:59:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 20 = '' 2024-05-17T14:59:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:59:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T14:59:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:59:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:59:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:59:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:59:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:59:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T14:59:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:59:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T14:59:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T14:59:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T14:59:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T14:59:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 20 = '' 2024-05-17T14:59:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T14:59:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T14:59:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:59:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:59:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T14:59:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T14:59:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T14:59:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T14:59:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=20 2024-05-17T14:59:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 20 = 20 2024-05-17T14:59:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:59:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-05-17T14:59:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:59:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap6 2024-05-17T14:59:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T14:59:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T14:59:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:59:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-05-17T14:59:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'test': 2024-05-17T14:59:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 512 MiB in 128 objects 2024-05-17T14:59:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T14:59:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 12 2024-05-17T14:59:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10bd697942f 2024-05-17T14:59:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10bd697942f 2024-05-17T14:59:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T14:59:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T14:59:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T14:59:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T14:59:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 14:54:18 2024 2024-05-17T14:59:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 14:58:52 2024 2024-05-17T14:59:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 14:54:18 2024 2024-05-17T14:59:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T14:59:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T14:59:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T14:59:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T14:59:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T14:59:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:59:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snap5-clean ']' 2024-05-17T14:59:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap5-clean 2024-05-17T14:59:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap5-clean 2024-05-17T14:59:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T14:59:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:59:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:59:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:59:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:59:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:59:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:59:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:59:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:59:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:59:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:59:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap6 2024-05-17T14:59:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap6 2024-05-17T14:59:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T14:59:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:59:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:59:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:59:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T14:59:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:59:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:59:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T14:59:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:59:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T14:59:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:59:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ clean_snap_name=snap6-clean 2024-05-17T14:59:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-05-17T14:59:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:59:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap6-clean 2024-05-17T14:59:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-05-17T14:59:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T14:59:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror test 2024-05-17T14:59:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:59:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T14:59:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=8 2024-05-17T14:59:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T14:59:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-17T14:59:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 7500800 bytes 2024-05-17T14:59:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 47834112 bytes 2024-05-17T14:59:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 75801600 bytes 2024-05-17T14:59:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 107876352 bytes 2024-05-17T14:59:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 168699904 bytes 2024-05-17T14:59:46.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 184121344 bytes 2024-05-17T14:59:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 214788096 bytes 2024-05-17T14:59:46.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 283639808 bytes 2024-05-17T14:59:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 307238912 bytes 2024-05-17T14:59:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 333906944 bytes 2024-05-17T14:59:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 406775808 bytes 2024-05-17T14:59:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 410215424 bytes 2024-05-17T14:59:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 440058880 bytes 2024-05-17T14:59:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 493870080 bytes 2024-05-17T14:59:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 527924224 bytes 2024-05-17T14:59:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 566510592 bytes 2024-05-17T14:59:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 687382528 bytes 2024-05-17T14:59:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 750358528 bytes 2024-05-17T14:59:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 791724032 bytes 2024-05-17T14:59:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 883249152 bytes 2024-05-17T14:59:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 936500224 bytes 2024-05-17T14:59:48.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 977253376 bytes 2024-05-17T14:59:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1093983232 bytes 2024-05-17T14:59:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1141006336 bytes 2024-05-17T14:59:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1166885888 bytes 2024-05-17T14:59:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1212437504 bytes 2024-05-17T14:59:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1264331776 bytes 2024-05-17T14:59:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1296771072 bytes 2024-05-17T14:59:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1323812864 bytes 2024-05-17T14:59:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1352131584 bytes 2024-05-17T14:59:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1357870080 bytes 2024-05-17T14:59:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1383086080 bytes 2024-05-17T14:59:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1410584576 bytes 2024-05-17T14:59:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1455395840 bytes 2024-05-17T14:59:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1480020992 bytes 2024-05-17T14:59:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1493671936 bytes 2024-05-17T14:59:50.113 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1536262144 bytes 2024-05-17T14:59:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1572705280 bytes 2024-05-17T14:59:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1641280512 bytes 2024-05-17T14:59:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1673649152 bytes 2024-05-17T14:59:50.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1749977088 bytes 2024-05-17T14:59:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1814012928 bytes 2024-05-17T14:59:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1814618112 bytes 2024-05-17T14:59:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1885970432 bytes 2024-05-17T14:59:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1922560000 bytes 2024-05-17T14:59:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1992992768 bytes 2024-05-17T14:59:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1993625600 bytes 2024-05-17T14:59:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2014519296 bytes 2024-05-17T14:59:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2059662336 bytes 2024-05-17T14:59:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2081478656 bytes 2024-05-17T14:59:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2159557632 bytes 2024-05-17T14:59:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2233845760 bytes 2024-05-17T14:59:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2301518848 bytes 2024-05-17T14:59:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2321249280 bytes 2024-05-17T14:59:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2346929152 bytes 2024-05-17T14:59:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2374503424 bytes 2024-05-17T14:59:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2460378112 bytes 2024-05-17T14:59:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2553336832 bytes 2024-05-17T14:59:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2590124032 bytes 2024-05-17T14:59:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2636900352 bytes 2024-05-17T14:59:53.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2676415488 bytes 2024-05-17T14:59:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2725695488 bytes 2024-05-17T14:59:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2822462464 bytes 2024-05-17T14:59:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2823064576 bytes 2024-05-17T14:59:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2844846080 bytes 2024-05-17T14:59:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2864324608 bytes 2024-05-17T14:59:53.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2895441920 bytes 2024-05-17T14:59:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T14:59:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T14:59:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T14:59:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:59:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-17T14:59:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:59:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:59:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T14:59:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T14:59:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T14:59:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T14:59:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-17T14:59:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:59:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:59:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T14:59:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T14:59:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T14:59:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T14:59:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-17T14:59:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='test: 2024-05-17T14:59:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: global_id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T14:59:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+replaying 2024-05-17T14:59:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: replaying, {"bytes_per_second":17572932.27,"bytes_per_snapshot":525707264.0,"last_snapshot_bytes":527187968,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957976,"remote_snapshot_timestamp":1715957976,"replay_state":"idle"} 2024-05-17T14:59:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: service: mirror.1 on smithi195 2024-05-17T14:59:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 14:59:46 2024-05-17T14:59:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: peer_sites: 2024-05-17T14:59:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: name: cluster2 2024-05-17T14:59:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+stopped 2024-05-17T14:59:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: local image is primary 2024-05-17T14:59:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 14:59:47' 2024-05-17T14:59:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ state: .*up+replaying' 2024-05-17T14:59:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T14:59:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ description: .*' 2024-05-17T14:59:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":17572932.27,"bytes_per_snapshot":525707264.0,"last_snapshot_bytes":527187968,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957976,"remote_snapshot_timestamp":1715957976,"replay_state":"idle"} 2024-05-17T14:59:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T14:59:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T14:59:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^up+' 2024-05-17T14:59:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T14:59:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep 'service: *mirror..* on ' 2024-05-17T14:59:54.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.1 on smithi195 2024-05-17T14:59:54.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update 2024-05-17T14:59:54.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-17T14:59:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 14:59:46' 2024-05-17T14:59:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-17 14:59:46' 2024-05-17T14:59:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T14:59:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T14:59:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local 'prev_last_update=2024-05-17 14:59:46' 2024-05-17T14:59:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T14:59:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T14:59:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-17T14:59:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='OKOKOK142144288mirror.1smithi19519.0.0-3735-gee59d86atrueOK42184296mirror.2smithi19519.0.0-3735-gee59d86afalseOK42224307mirror.3smithi19519.0.0-3735-gee59d86afalseOK44024405mirror.0smithi19519.0.0-3735-gee59d86afalseOKtest72894289-d917-4c92-922e-8f6ade3b96caup+replayingreplaying, {"bytes_per_second":17572932.27,"bytes_per_snapshot":525707264.0,"last_snapshot_bytes":527187968,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715957976,"remote_snapshot_timestamp":1715957976,"replay_state":"idle"}42144288mirror.1smithi1952024-05-17 14:59:46cluster27e816122-c11e-4a54-9bab-7372f406213bup+stoppedlocal image is primary2024-05-17 14:59:47' 2024-05-17T14:59:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update state 2024-05-17T14:59:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-17T14:59:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 14:59:46' 2024-05-17T14:59:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-17T14:59:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ state=up+replaying 2024-05-17T14:59:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T14:59:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep up+replaying 2024-05-17T14:59:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T14:59:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:59:54.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T14:59:54.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap7 2024-05-17T14:59:54.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap7 2024-05-17T14:59:54.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T14:59:54.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T14:59:54.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T14:59:54.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap7 2024-05-17T14:59:54.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:00:08.735 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9853368059861926) > weight (0.25) 2024-05-17T15:00:08.735 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-05-17T15:00:08.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-05-17T15:00:08.736 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-05-17T15:00:08.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:00:08.736+0000 7f0c87cbf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 37991) UID: 1000 2024-05-17T15:00:15.737 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-05-17T15:00:15.737 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:00:15.738 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-05-17T15:00:15.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-05-17T15:00:15.738 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-05-17T15:00:15.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-05-17T15:00:15.742 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2024-05-17T15:00:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-17T15:00:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:00:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:00:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:00:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-17T15:00:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:00:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:00:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:00:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:00:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:00:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:00:22.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:00:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-17T15:00:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:00:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:00:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:00:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:00:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:00:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:00:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-17T15:00:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:00:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:00:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:00:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:00:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:00:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:00:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:00:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:00:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:00:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T15:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:00:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:00:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:00:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:00:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:00:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:00:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:00:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:00:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:00:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:00:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:00:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T15:00:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T15:00:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:00:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:00:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:00:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:23.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:00:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:00:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:00:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:00:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:00:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:00:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:00:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:00:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:00:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:00:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:00:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:00:23.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:00:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T15:00:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T15:00:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:00:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:00:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:00:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:00:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:00:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:23.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:00:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:00:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:00:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:00:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:00:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:00:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:00:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:00:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:00:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:00:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:00:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T15:00:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-05-17T15:00:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:00:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:00:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:00:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:00:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:00:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:00:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:00:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:00:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:00:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:00:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:00:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:00:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:00:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:00:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:00:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:00:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:00:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:00:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:00:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-05-17T15:00:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:00:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 1 2024-05-17T15:00:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:00:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:00:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T15:00:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:00:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:00:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:00:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:00:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:00:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:00:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:00:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:00:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:00:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:00:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:00:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:00:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:00:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:00:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:00:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:00:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T15:00:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T15:00:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:00:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:00:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:00:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:00:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:00:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:00:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:00:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:00:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:00:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:00:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:00:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:00:25.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:00:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:00:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:00:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:00:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T15:00:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T15:00:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:00:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:00:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:00:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:00:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:00:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:00:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:00:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:00:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:00:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:00:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:00:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:00:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:00:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:00:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:00:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:00:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T15:00:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-05-17T15:00:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:00:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:00:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:00:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:00:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:00:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:00:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:00:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:00:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:00:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:00:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:00:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:00:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:00:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:00:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:00:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:00:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:00:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:00:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:00:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-05-17T15:00:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:00:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 2 2024-05-17T15:00:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:00:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:00:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T15:00:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:00:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:00:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:00:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:00:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:00:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:00:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:00:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:00:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:00:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:00:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:00:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:00:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:00:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:00:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:00:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:00:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T15:00:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T15:00:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:00:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:00:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:00:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:00:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:00:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:00:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:00:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:00:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:00:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:00:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:00:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:27.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:00:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:00:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:00:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:00:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:00:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T15:00:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T15:00:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:00:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:00:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:00:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:00:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:00:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:00:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:00:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:00:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:00:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:00:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:00:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:00:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:00:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:00:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:00:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:00:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T15:00:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-05-17T15:00:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:00:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:00:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:00:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:00:27.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:00:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:00:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:00:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:00:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:00:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:00:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:00:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:00:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:00:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:00:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:00:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:00:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:00:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:00:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:00:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-05-17T15:00:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:00:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 4 2024-05-17T15:00:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:00:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:00:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T15:00:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:00:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:00:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:00:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:00:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:00:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:00:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:00:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:00:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:00:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:00:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:00:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:00:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:00:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:00:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:00:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:00:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T15:00:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T15:00:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:00:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:00:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:00:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:00:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:00:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:32.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:32.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:00:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:00:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:00:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:00:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:00:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:00:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:00:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:00:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:00:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:00:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:00:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T15:00:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T15:00:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:00:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:00:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:00:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:00:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:00:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:00:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:00:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:00:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:00:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:00:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:00:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:00:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:00:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:00:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:00:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:00:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T15:00:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-05-17T15:00:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:00:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:00:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:00:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:00:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:00:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:00:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:00:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:00:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:00:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:00:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:00:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:00:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:00:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:00:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:00:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:00:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:00:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:00:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:00:32.275 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-05-17T15:00:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:00:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 8 2024-05-17T15:00:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:00:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:00:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T15:00:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:00:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:00:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:00:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:00:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:00:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:00:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:00:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:00:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:00:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:00:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:00:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:00:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:00:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:00:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:00:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:00:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T15:00:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T15:00:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:00:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:00:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:00:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:00:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:00:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:00:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:00:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:00:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T15:00:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T15:00:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:00:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:00:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:00:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:00:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:00:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:00:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:00:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:00:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:00:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:00:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:00:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:00:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:00:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:00:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:00:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:00:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T15:00:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-05-17T15:00:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:00:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:00:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:00:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:00:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:00:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:00:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:00:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:00:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:00:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:00:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:00:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:00:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:00:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:00:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:00:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:00:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:00:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:00:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:00:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-05-17T15:00:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:00:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 8 2024-05-17T15:00:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:00:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:00:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T15:00:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:00:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:00:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:00:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:00:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:00:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:00:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:00:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:00:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:00:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:00:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:00:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:00:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:00:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:00:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:00:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:00:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:00:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T15:00:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T15:00:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:00:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:00:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:00:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:00:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:00:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:00:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:00:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:00:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:00:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:00:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:00:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:00:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:00:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:00:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:00:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:00:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:00:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T15:00:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:00:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:00:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:00:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:00:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T15:00:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:00:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:00:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:00:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:00:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:00:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:00:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:00:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:00:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:00:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:00:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:00:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:00:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:00:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:00:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:00:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:00:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:00:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:00:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:00:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:00:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:00:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:00:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:00:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:00:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:00:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:00:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:00:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:00:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:00:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:00:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T15:00:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:00:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:00:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:00:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:00:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:00:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:00:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:00:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:00:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-17T15:00:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:00:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:00:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:00:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:00:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:00:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:00:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-05-17T15:00:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:00:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:00:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:00:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:00:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-test.status 2024-05-17T15:00:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-test.status 2024-05-17T15:00:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:00:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T15:00:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-test.status 2024-05-17T15:00:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-test.status 2024-05-17T15:00:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:00:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:00:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-05-17T15:00:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:00:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:00:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:00:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-05-17T15:00:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 23 2024-05-17T15:00:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T15:00:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:00:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:00:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:00:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T15:00:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:00:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-05-17T15:00:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:00:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=23 2024-05-17T15:00:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:00:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:00:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:00:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:00:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:00:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:00:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:00:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:00:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:00:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:00:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:00:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:00:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:00:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:00:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:00:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:00:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 23 = '' 2024-05-17T15:00:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:00:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:00:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:00:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:00:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:00:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:00:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:00:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:00:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:00:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:00:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:00:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:00:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:00:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:00:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:00:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 23 = '' 2024-05-17T15:00:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:00:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:00:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:00:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:00:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:00:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:00:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:00:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:00:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:00:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:00:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:00:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:00:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:00:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:00:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:00:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 23 = '' 2024-05-17T15:00:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:00:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:00:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:00:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:00:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:00:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:00:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:00:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:00:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:00:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:00:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=23 2024-05-17T15:00:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 23 = 23 2024-05-17T15:00:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:00:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-05-17T15:00:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:00:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:00:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:00:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap7 2024-05-17T15:00:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:00:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:00:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:00:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-05-17T15:00:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'test': 2024-05-17T15:00:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 512 MiB in 128 objects 2024-05-17T15:00:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:00:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 14 2024-05-17T15:00:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10bd697942f 2024-05-17T15:00:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10bd697942f 2024-05-17T15:00:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:00:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:00:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:00:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:00:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 14:54:18 2024 2024-05-17T15:00:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 14:58:52 2024 2024-05-17T15:00:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 14:54:18 2024 2024-05-17T15:00:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:00:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:00:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:00:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T15:00:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:00:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:00:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snap6-clean ']' 2024-05-17T15:00:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap6-clean 2024-05-17T15:00:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:00:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:00:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap6-clean 2024-05-17T15:00:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:00:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:00:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:00:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:00:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:00:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:00:55.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:00:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:00:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:00:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:00:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:00:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap7 2024-05-17T15:00:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:00:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:00:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap7 2024-05-17T15:00:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:00:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:00:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:00:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:00:56.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:00:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:00:57.433 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:00:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:00:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:00:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:00:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:00:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ clean_snap_name=snap7-clean 2024-05-17T15:00:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-05-17T15:00:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:00:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:00:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:00:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap7-clean 2024-05-17T15:00:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-05-17T15:00:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:00:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror test 2024-05-17T15:00:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:00:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:00:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:00:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:00:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=9 2024-05-17T15:00:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:00:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-17T15:00:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 79691776 bytes 2024-05-17T15:00:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 125376512 bytes 2024-05-17T15:00:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 161969152 bytes 2024-05-17T15:00:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 238915584 bytes 2024-05-17T15:00:59.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 276308992 bytes 2024-05-17T15:00:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 302685184 bytes 2024-05-17T15:00:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 323962880 bytes 2024-05-17T15:00:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 384584704 bytes 2024-05-17T15:01:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 486425600 bytes 2024-05-17T15:01:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 546585600 bytes 2024-05-17T15:01:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 585482240 bytes 2024-05-17T15:01:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 658625536 bytes 2024-05-17T15:01:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 720078848 bytes 2024-05-17T15:01:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 726212608 bytes 2024-05-17T15:01:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 740493312 bytes 2024-05-17T15:01:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 786173952 bytes 2024-05-17T15:01:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 895647744 bytes 2024-05-17T15:01:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 902672384 bytes 2024-05-17T15:01:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 953486336 bytes 2024-05-17T15:01:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 1033121792 bytes 2024-05-17T15:01:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 1071097856 bytes 2024-05-17T15:01:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1140705280 bytes 2024-05-17T15:01:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1165914112 bytes 2024-05-17T15:01:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1197960192 bytes 2024-05-17T15:01:02.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1218001920 bytes 2024-05-17T15:01:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1279528960 bytes 2024-05-17T15:01:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1344519168 bytes 2024-05-17T15:01:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1380970496 bytes 2024-05-17T15:01:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1447012352 bytes 2024-05-17T15:01:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1492156416 bytes 2024-05-17T15:01:02.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1516658688 bytes 2024-05-17T15:01:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1573258240 bytes 2024-05-17T15:01:03.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1609873408 bytes 2024-05-17T15:01:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1634053120 bytes 2024-05-17T15:01:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1684067328 bytes 2024-05-17T15:01:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1722382336 bytes 2024-05-17T15:01:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1802398720 bytes 2024-05-17T15:01:03.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1839040512 bytes 2024-05-17T15:01:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1961671680 bytes 2024-05-17T15:01:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1974966272 bytes 2024-05-17T15:01:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 2029846528 bytes 2024-05-17T15:01:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 2059448320 bytes 2024-05-17T15:01:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 2068555776 bytes 2024-05-17T15:01:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 2070441984 bytes 2024-05-17T15:01:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2167154688 bytes 2024-05-17T15:01:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2207039488 bytes 2024-05-17T15:01:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2247164928 bytes 2024-05-17T15:01:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2326697984 bytes 2024-05-17T15:01:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2385102848 bytes 2024-05-17T15:01:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2449137664 bytes 2024-05-17T15:01:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2461348864 bytes 2024-05-17T15:01:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2527080448 bytes 2024-05-17T15:01:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2550559744 bytes 2024-05-17T15:01:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2551138304 bytes 2024-05-17T15:01:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2558712832 bytes 2024-05-17T15:01:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2560574464 bytes 2024-05-17T15:01:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2561198080 bytes 2024-05-17T15:01:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2607671296 bytes 2024-05-17T15:01:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2676404224 bytes 2024-05-17T15:01:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2722585600 bytes 2024-05-17T15:01:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2773472256 bytes 2024-05-17T15:01:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2795791360 bytes 2024-05-17T15:01:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2816709632 bytes 2024-05-17T15:01:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2848218112 bytes 2024-05-17T15:01:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2919638016 bytes 2024-05-17T15:01:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2944112640 bytes 2024-05-17T15:01:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2979952640 bytes 2024-05-17T15:01:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2980560896 bytes 2024-05-17T15:01:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 3021722624 bytes 2024-05-17T15:01:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 3058001920 bytes 2024-05-17T15:01:07.347 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3136478208 bytes 2024-05-17T15:01:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3146352640 bytes 2024-05-17T15:01:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3188439040 bytes 2024-05-17T15:01:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3260561408 bytes 2024-05-17T15:01:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:01:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:01:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:01:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:01:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-17T15:01:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:01:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T15:01:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T15:01:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T15:01:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:01:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:01:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-17T15:01:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:01:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T15:01:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T15:01:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T15:01:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T15:01:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T15:01:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-17T15:01:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='test: 2024-05-17T15:01:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: global_id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T15:01:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+replaying 2024-05-17T15:01:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715957976,"remote_snapshot_timestamp":1715957976,"replay_state":"idle"} 2024-05-17T15:01:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: service: mirror.2 on smithi195 2024-05-17T15:01:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 15:00:46 2024-05-17T15:01:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: peer_sites: 2024-05-17T15:01:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: name: cluster2 2024-05-17T15:01:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+stopped 2024-05-17T15:01:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: local image is primary 2024-05-17T15:01:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 15:00:47' 2024-05-17T15:01:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ state: .*up+replaying' 2024-05-17T15:01:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:01:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ description: .*' 2024-05-17T15:01:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715957976,"remote_snapshot_timestamp":1715957976,"replay_state":"idle"} 2024-05-17T15:01:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:01:08.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T15:01:08.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^up+' 2024-05-17T15:01:08.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T15:01:08.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep 'service: *mirror..* on ' 2024-05-17T15:01:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.2 on smithi195 2024-05-17T15:01:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update 2024-05-17T15:01:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-17T15:01:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 15:00:46' 2024-05-17T15:01:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-17 15:00:46' 2024-05-17T15:01:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:01:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T15:01:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local 'prev_last_update=2024-05-17 15:00:46' 2024-05-17T15:01:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T15:01:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T15:01:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-17T15:01:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='OKOKOK142184296mirror.2smithi19519.0.0-3735-gee59d86atrueOK42224307mirror.3smithi19519.0.0-3735-gee59d86afalseOK44024405mirror.0smithi19519.0.0-3735-gee59d86afalseOK45314534mirror.1smithi19519.0.0-3735-gee59d86afalseOKtest72894289-d917-4c92-922e-8f6ade3b96caup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715957976,"remote_snapshot_timestamp":1715957976,"replay_state":"idle"}42184296mirror.2smithi1952024-05-17 15:00:46cluster27e816122-c11e-4a54-9bab-7372f406213bup+stoppedlocal image is primary2024-05-17 15:00:47' 2024-05-17T15:01:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update state 2024-05-17T15:01:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-17T15:01:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 15:00:46' 2024-05-17T15:01:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-17T15:01:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ state=up+replaying 2024-05-17T15:01:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T15:01:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep up+replaying 2024-05-17T15:01:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T15:01:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:01:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:01:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap8 2024-05-17T15:01:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap8 2024-05-17T15:01:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:01:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap8 2024-05-17T15:01:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:01:14.850 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5823609355598022) > weight (0.25) 2024-05-17T15:01:14.851 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9976388735249214) > weight (0.25) 2024-05-17T15:01:14.851 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.46676067284487255) > weight (0.25) 2024-05-17T15:01:14.851 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7133108769620101) > weight (0.25) 2024-05-17T15:01:14.851 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2024-05-17T15:01:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-17T15:01:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:01:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-17T15:01:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:01:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:01:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:01:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:01:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:01:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-17T15:01:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:01:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:01:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:01:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:01:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-17T15:01:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:01:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:01:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:01:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:01:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:01:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:01:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:01:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:01:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:01:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:01:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:01:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T15:01:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:01:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:01:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:01:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:01:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:01:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:01:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:01:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:01:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:01:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:01:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:01:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:01:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:01:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:01:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:01:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:01:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:01:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:01:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:01:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:01:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:01:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:01:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:01:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:01:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:01:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:01:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:01:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:01:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:01:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:01:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:01:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:01:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:01:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T15:01:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:01:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:01:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:01:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:01:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T15:01:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:01:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:01:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:01:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:01:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:01:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:01:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:01:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:01:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:01:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:01:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:01:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:01:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:01:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:01:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:01:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:01:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:01:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:01:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:01:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:01:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:01:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:01:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:01:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:01:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:01:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:01:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:01:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:01:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:01:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:01:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:01:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:01:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:01:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T15:01:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:01:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:01:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:01:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:01:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T15:01:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:01:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:01:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:01:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:01:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:01:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:01:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:01:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:01:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:01:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:01:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:01:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:01:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:01:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:01:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:01:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:01:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:01:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:01:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:01:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:01:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:01:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:01:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:01:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:01:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:01:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:01:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:01:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:01:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:01:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:01:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:01:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:01:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:01:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T15:01:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:01:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:01:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:01:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:01:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:01:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:01:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:01:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:01:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-17T15:01:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:01:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:01:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:01:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:01:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-05-17T15:01:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:01:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:01:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-test.status 2024-05-17T15:01:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-test.status 2024-05-17T15:01:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:01:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T15:01:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-test.status 2024-05-17T15:01:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-test.status 2024-05-17T15:01:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:01:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:01:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-05-17T15:01:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:01:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-05-17T15:01:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 26 2024-05-17T15:01:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T15:01:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:01:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T15:01:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-05-17T15:01:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:01:41.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:01:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=26 2024-05-17T15:01:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:01:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:01:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:01:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:01:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:01:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:01:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:01:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:01:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:01:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:01:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:01:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:01:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:01:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:01:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 26 = '' 2024-05-17T15:01:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:01:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:01:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:01:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:01:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:01:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:01:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:01:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:01:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:01:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:01:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:01:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:01:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:01:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 26 = '' 2024-05-17T15:01:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:01:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:01:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:01:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:01:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:01:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:01:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:01:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:01:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:01:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:01:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:01:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:01:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:01:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 26 = '' 2024-05-17T15:01:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:01:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:01:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:01:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:01:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:01:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:01:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:01:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:01:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=26 2024-05-17T15:01:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 26 = 26 2024-05-17T15:01:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:01:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-05-17T15:01:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:01:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap8 2024-05-17T15:01:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:01:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:01:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:01:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-05-17T15:01:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'test': 2024-05-17T15:01:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 512 MiB in 128 objects 2024-05-17T15:01:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:01:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 16 2024-05-17T15:01:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10bd697942f 2024-05-17T15:01:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10bd697942f 2024-05-17T15:01:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:01:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:01:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:01:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:01:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 14:54:18 2024 2024-05-17T15:01:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:00:55 2024 2024-05-17T15:01:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 14:54:18 2024 2024-05-17T15:01:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:01:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:01:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:01:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T15:01:46.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:01:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:01:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snap7-clean ']' 2024-05-17T15:01:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap7-clean 2024-05-17T15:01:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap7-clean 2024-05-17T15:01:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:01:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:01:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:01:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:01:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:01:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:01:46.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:01:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:01:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:01:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:01:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:01:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap8 2024-05-17T15:01:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap8 2024-05-17T15:01:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:01:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:01:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:01:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:01:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:01:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:01:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:01:49.280 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:01:49.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:01:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:01:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:01:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ clean_snap_name=snap8-clean 2024-05-17T15:01:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-05-17T15:01:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:01:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap8-clean 2024-05-17T15:01:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-05-17T15:01:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:01:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror test 2024-05-17T15:01:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:01:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:01:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:01:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:01:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=14 2024-05-17T15:01:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:01:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-17T15:01:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 60745728 bytes 2024-05-17T15:01:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 96706560 bytes 2024-05-17T15:01:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 170744832 bytes 2024-05-17T15:01:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 184559616 bytes 2024-05-17T15:01:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 225830912 bytes 2024-05-17T15:01:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 264144896 bytes 2024-05-17T15:01:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 315588608 bytes 2024-05-17T15:01:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 353424384 bytes 2024-05-17T15:01:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 354029568 bytes 2024-05-17T15:01:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 410843136 bytes 2024-05-17T15:01:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 447627264 bytes 2024-05-17T15:01:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 524340224 bytes 2024-05-17T15:01:51.498 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 565506048 bytes 2024-05-17T15:01:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 567263232 bytes 2024-05-17T15:01:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 584498176 bytes 2024-05-17T15:01:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 625393664 bytes 2024-05-17T15:01:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 682180608 bytes 2024-05-17T15:01:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 729884672 bytes 2024-05-17T15:01:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 802397184 bytes 2024-05-17T15:01:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 833988608 bytes 2024-05-17T15:01:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 875633664 bytes 2024-05-17T15:01:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 913629184 bytes 2024-05-17T15:01:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 979687424 bytes 2024-05-17T15:01:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1022678016 bytes 2024-05-17T15:01:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1029426176 bytes 2024-05-17T15:01:52.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1030036480 bytes 2024-05-17T15:01:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1076397056 bytes 2024-05-17T15:01:53.121 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1142154240 bytes 2024-05-17T15:01:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1179037696 bytes 2024-05-17T15:01:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1252888576 bytes 2024-05-17T15:01:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1295035392 bytes 2024-05-17T15:01:53.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1295760384 bytes 2024-05-17T15:01:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1341005824 bytes 2024-05-17T15:01:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1415475200 bytes 2024-05-17T15:01:54.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1495775232 bytes 2024-05-17T15:01:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1532832768 bytes 2024-05-17T15:01:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1590497280 bytes 2024-05-17T15:01:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1673393152 bytes 2024-05-17T15:01:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1722566656 bytes 2024-05-17T15:01:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1789256704 bytes 2024-05-17T15:01:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1811489792 bytes 2024-05-17T15:01:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1853589504 bytes 2024-05-17T15:01:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1887006720 bytes 2024-05-17T15:01:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1937518592 bytes 2024-05-17T15:01:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1955768320 bytes 2024-05-17T15:01:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1992057856 bytes 2024-05-17T15:01:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2084137984 bytes 2024-05-17T15:01:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2188119040 bytes 2024-05-17T15:01:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2188744704 bytes 2024-05-17T15:01:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2217014272 bytes 2024-05-17T15:01:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2244649984 bytes 2024-05-17T15:01:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2293609472 bytes 2024-05-17T15:01:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2354008064 bytes 2024-05-17T15:01:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2402809856 bytes 2024-05-17T15:01:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2505107456 bytes 2024-05-17T15:01:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2559930368 bytes 2024-05-17T15:01:56.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2575471616 bytes 2024-05-17T15:01:56.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2589450240 bytes 2024-05-17T15:01:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2598438912 bytes 2024-05-17T15:01:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2619481088 bytes 2024-05-17T15:01:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2662350848 bytes 2024-05-17T15:01:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2712868864 bytes 2024-05-17T15:01:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2774311936 bytes 2024-05-17T15:01:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2781236224 bytes 2024-05-17T15:01:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2815555584 bytes 2024-05-17T15:01:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2832175104 bytes 2024-05-17T15:01:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2868295680 bytes 2024-05-17T15:01:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2895751168 bytes 2024-05-17T15:01:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2971932672 bytes 2024-05-17T15:01:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 2998874112 bytes 2024-05-17T15:01:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3027650560 bytes 2024-05-17T15:01:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3076721664 bytes 2024-05-17T15:01:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3102700544 bytes 2024-05-17T15:01:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3103305728 bytes 2024-05-17T15:01:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3136221184 bytes 2024-05-17T15:01:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3162552320 bytes 2024-05-17T15:01:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3193205760 bytes 2024-05-17T15:01:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3247304704 bytes 2024-05-17T15:01:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3247907840 bytes 2024-05-17T15:01:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3299826688 bytes 2024-05-17T15:01:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3400232960 bytes 2024-05-17T15:01:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3424945152 bytes 2024-05-17T15:01:59.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3518583808 bytes 2024-05-17T15:01:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3537479680 bytes 2024-05-17T15:01:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3561856000 bytes 2024-05-17T15:01:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3643661312 bytes 2024-05-17T15:02:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3713035264 bytes 2024-05-17T15:02:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3714924544 bytes 2024-05-17T15:02:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3755211776 bytes 2024-05-17T15:02:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3794253824 bytes 2024-05-17T15:02:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 3859512320 bytes 2024-05-17T15:02:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9200 IOs, 3918906368 bytes 2024-05-17T15:02:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9300 IOs, 4014981120 bytes 2024-05-17T15:02:01.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9400 IOs, 4059354112 bytes 2024-05-17T15:02:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9500 IOs, 4186947584 bytes 2024-05-17T15:02:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9600 IOs, 4247652352 bytes 2024-05-17T15:02:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9700 IOs, 4297919488 bytes 2024-05-17T15:02:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9800 IOs, 4304651264 bytes 2024-05-17T15:02:01.742 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.756652745129753) > weight (0.25) 2024-05-17T15:02:01.742 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4872366646644042) > weight (0.25) 2024-05-17T15:02:01.743 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.33376233674619216) > weight (0.25) 2024-05-17T15:02:01.743 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9019624042116989) > weight (0.25) 2024-05-17T15:02:01.743 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2024-05-17T15:02:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9900 IOs, 4352611328 bytes 2024-05-17T15:02:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10000 IOs, 4416380928 bytes 2024-05-17T15:02:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10100 IOs, 4424949760 bytes 2024-05-17T15:02:02.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10200 IOs, 4449154048 bytes 2024-05-17T15:02:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10300 IOs, 4487865344 bytes 2024-05-17T15:02:02.441 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10400 IOs, 4549439488 bytes 2024-05-17T15:02:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10500 IOs, 4597237760 bytes 2024-05-17T15:02:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10600 IOs, 4657054720 bytes 2024-05-17T15:02:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10700 IOs, 4741178368 bytes 2024-05-17T15:02:03.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10800 IOs, 4790020096 bytes 2024-05-17T15:02:03.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10900 IOs, 4830372864 bytes 2024-05-17T15:02:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11000 IOs, 4841891840 bytes 2024-05-17T15:02:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11100 IOs, 4900478976 bytes 2024-05-17T15:02:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11200 IOs, 4931609600 bytes 2024-05-17T15:02:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11300 IOs, 4932231168 bytes 2024-05-17T15:02:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11400 IOs, 4932868096 bytes 2024-05-17T15:02:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11500 IOs, 4956743680 bytes 2024-05-17T15:02:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:02:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:02:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:02:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:02:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-17T15:02:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:02:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T15:02:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T15:02:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T15:02:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:02:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:02:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-17T15:02:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:02:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T15:02:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T15:02:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T15:02:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T15:02:04.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T15:02:04.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-17T15:02:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='test: 2024-05-17T15:02:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: global_id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T15:02:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+replaying 2024-05-17T15:02:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: replaying, {"bytes_per_second":17833130.67,"bytes_per_snapshot":533725184.0,"last_snapshot_bytes":534993920,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715958100,"remote_snapshot_timestamp":1715958100,"replay_state":"idle"} 2024-05-17T15:02:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: service: mirror.2 on smithi195 2024-05-17T15:02:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 15:01:46 2024-05-17T15:02:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: peer_sites: 2024-05-17T15:02:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: name: cluster2 2024-05-17T15:02:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+stopped 2024-05-17T15:02:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: local image is primary 2024-05-17T15:02:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 15:01:47' 2024-05-17T15:02:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ state: .*up+replaying' 2024-05-17T15:02:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:02:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ description: .*' 2024-05-17T15:02:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":17833130.67,"bytes_per_snapshot":533725184.0,"last_snapshot_bytes":534993920,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715958100,"remote_snapshot_timestamp":1715958100,"replay_state":"idle"} 2024-05-17T15:02:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:02:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T15:02:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^up+' 2024-05-17T15:02:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T15:02:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep 'service: *mirror..* on ' 2024-05-17T15:02:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.2 on smithi195 2024-05-17T15:02:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update 2024-05-17T15:02:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-17T15:02:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 15:01:46' 2024-05-17T15:02:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-17 15:01:46' 2024-05-17T15:02:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:02:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T15:02:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local 'prev_last_update=2024-05-17 15:01:46' 2024-05-17T15:02:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T15:02:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T15:02:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-17T15:02:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='OKOKOK142184296mirror.2smithi19519.0.0-3735-gee59d86atrueOK42224307mirror.3smithi19519.0.0-3735-gee59d86afalseOK44024405mirror.0smithi19519.0.0-3735-gee59d86afalseOK45314534mirror.1smithi19519.0.0-3735-gee59d86afalseOKtest72894289-d917-4c92-922e-8f6ade3b96caup+replayingreplaying, {"bytes_per_second":17833130.67,"bytes_per_snapshot":533725184.0,"last_snapshot_bytes":534993920,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715958100,"remote_snapshot_timestamp":1715958100,"replay_state":"idle"}42184296mirror.2smithi1952024-05-17 15:01:46cluster27e816122-c11e-4a54-9bab-7372f406213bup+stoppedlocal image is primary2024-05-17 15:01:47' 2024-05-17T15:02:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update state 2024-05-17T15:02:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-17T15:02:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 15:01:46' 2024-05-17T15:02:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-17T15:02:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ state=up+replaying 2024-05-17T15:02:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T15:02:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep up+replaying 2024-05-17T15:02:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T15:02:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:02:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:02:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap9 2024-05-17T15:02:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap9 2024-05-17T15:02:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:02:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap9 2024-05-17T15:02:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:02:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-17T15:02:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:02:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-17T15:02:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:02:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:02:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:02:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:02:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-17T15:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:02:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:02:36.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:02:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-17T15:02:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:02:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:02:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:02:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:02:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:02:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:02:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:02:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:02:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:02:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:02:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:02:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T15:02:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:02:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:02:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:02:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:02:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:02:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:02:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:02:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:02:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:02:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:02:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:02:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:02:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:02:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:02:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:02:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:02:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:02:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:02:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:02:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:02:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T15:02:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:02:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:02:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:02:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:02:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T15:02:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:02:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:02:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:02:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:02:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:02:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:02:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:02:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:02:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:02:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:02:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:02:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:02:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:02:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:02:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:02:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:02:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:02:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:02:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:02:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:02:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:02:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:02:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:02:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:02:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:02:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:02:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:02:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:02:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:02:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:02:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:02:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:02:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:02:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T15:02:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:02:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:02:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:02:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:02:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T15:02:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:02:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:02:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:02:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:02:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:02:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:02:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:02:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:02:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:02:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:02:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:02:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:02:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:02:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:02:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:02:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:02:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:02:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:02:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:02:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:02:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:02:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:02:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:02:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:02:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:02:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:02:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:02:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:02:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:02:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:02:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:02:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:02:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:02:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T15:02:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:02:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:02:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:02:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:02:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:02:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-17T15:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-05-17T15:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-test.status 2024-05-17T15:02:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-test.status 2024-05-17T15:02:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:02:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T15:02:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-test.status 2024-05-17T15:02:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-test.status 2024-05-17T15:02:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:02:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:02:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-05-17T15:02:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:02:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-05-17T15:02:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 29 2024-05-17T15:02:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T15:02:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:02:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T15:02:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:02:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-05-17T15:02:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:02:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=29 2024-05-17T15:02:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:02:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:02:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:02:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:02:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:02:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:02:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:02:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:02:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:02:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:02:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:02:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:02:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:02:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:02:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 29 = '' 2024-05-17T15:02:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:02:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:02:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:02:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:02:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:02:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:02:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:02:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:02:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:02:39.212 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:02:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:02:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:02:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:02:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 29 = '' 2024-05-17T15:02:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:02:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:02:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:02:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:02:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:02:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:02:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:02:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:02:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:02:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:02:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:02:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:02:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:02:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 29 = '' 2024-05-17T15:02:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:02:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:02:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:02:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:02:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:02:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:02:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:02:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:02:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=29 2024-05-17T15:02:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 29 = 29 2024-05-17T15:02:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:02:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-05-17T15:02:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:02:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap9 2024-05-17T15:02:41.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:02:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:02:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:02:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-05-17T15:02:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'test': 2024-05-17T15:02:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 512 MiB in 128 objects 2024-05-17T15:02:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:02:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 18 2024-05-17T15:02:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10bd697942f 2024-05-17T15:02:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10bd697942f 2024-05-17T15:02:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:02:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:02:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:02:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:02:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 14:54:18 2024 2024-05-17T15:02:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:00:55 2024 2024-05-17T15:02:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 14:54:18 2024 2024-05-17T15:02:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:02:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:02:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:02:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T15:02:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:02:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:02:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snap8-clean ']' 2024-05-17T15:02:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap8-clean 2024-05-17T15:02:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap8-clean 2024-05-17T15:02:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:02:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:02:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:02:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:02:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:02:43.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:02:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:02:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:02:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:02:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:02:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:02:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap9 2024-05-17T15:02:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap9 2024-05-17T15:02:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:02:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:02:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:02:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:02:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:02:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:02:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:02:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:02:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:02:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:02:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:02:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ clean_snap_name=snap9-clean 2024-05-17T15:02:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-05-17T15:02:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:02:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap9-clean 2024-05-17T15:02:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-05-17T15:02:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:02:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror test 2024-05-17T15:02:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:02:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:02:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=11 2024-05-17T15:02:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:02:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-05-17T15:02:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 64152576 bytes 2024-05-17T15:02:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 172020736 bytes 2024-05-17T15:02:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 172645376 bytes 2024-05-17T15:02:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 187488256 bytes 2024-05-17T15:02:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 224252928 bytes 2024-05-17T15:02:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 267637760 bytes 2024-05-17T15:02:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 291953664 bytes 2024-05-17T15:02:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 292576256 bytes 2024-05-17T15:02:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 312632320 bytes 2024-05-17T15:02:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 387816448 bytes 2024-05-17T15:02:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 408877056 bytes 2024-05-17T15:02:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 513727488 bytes 2024-05-17T15:02:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 595731456 bytes 2024-05-17T15:02:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 636028928 bytes 2024-05-17T15:02:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 672620544 bytes 2024-05-17T15:02:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 728162304 bytes 2024-05-17T15:02:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 745821184 bytes 2024-05-17T15:02:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 841363456 bytes 2024-05-17T15:02:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 937575424 bytes 2024-05-17T15:02:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 989213696 bytes 2024-05-17T15:02:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 992996352 bytes 2024-05-17T15:02:49.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1071610880 bytes 2024-05-17T15:02:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1083433984 bytes 2024-05-17T15:02:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1141492736 bytes 2024-05-17T15:02:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1171535872 bytes 2024-05-17T15:02:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1201345536 bytes 2024-05-17T15:02:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1258932224 bytes 2024-05-17T15:02:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1299963904 bytes 2024-05-17T15:02:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1317712896 bytes 2024-05-17T15:02:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1361742848 bytes 2024-05-17T15:02:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1407636480 bytes 2024-05-17T15:02:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1437381632 bytes 2024-05-17T15:02:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1462508544 bytes 2024-05-17T15:02:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1492820992 bytes 2024-05-17T15:02:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1566374912 bytes 2024-05-17T15:02:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1629696000 bytes 2024-05-17T15:02:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1697031168 bytes 2024-05-17T15:02:51.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1713142784 bytes 2024-05-17T15:02:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1778691072 bytes 2024-05-17T15:02:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1843608576 bytes 2024-05-17T15:02:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1904398336 bytes 2024-05-17T15:02:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1950249984 bytes 2024-05-17T15:02:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 2001879040 bytes 2024-05-17T15:02:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 2039879680 bytes 2024-05-17T15:02:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2054096896 bytes 2024-05-17T15:02:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2083415040 bytes 2024-05-17T15:02:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2158521344 bytes 2024-05-17T15:02:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2166876160 bytes 2024-05-17T15:02:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2211440640 bytes 2024-05-17T15:02:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2252991488 bytes 2024-05-17T15:02:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2312069120 bytes 2024-05-17T15:02:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2332207104 bytes 2024-05-17T15:02:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2415745024 bytes 2024-05-17T15:02:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2486541312 bytes 2024-05-17T15:02:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2511732736 bytes 2024-05-17T15:02:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2534695936 bytes 2024-05-17T15:02:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2558210048 bytes 2024-05-17T15:02:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2624273408 bytes 2024-05-17T15:02:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2662850560 bytes 2024-05-17T15:02:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2683947008 bytes 2024-05-17T15:02:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2738689024 bytes 2024-05-17T15:02:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2784070656 bytes 2024-05-17T15:02:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2803793920 bytes 2024-05-17T15:02:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2867173376 bytes 2024-05-17T15:02:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2887130112 bytes 2024-05-17T15:02:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2898660352 bytes 2024-05-17T15:02:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2948519936 bytes 2024-05-17T15:02:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 3021857792 bytes 2024-05-17T15:02:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 3075201024 bytes 2024-05-17T15:02:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 3075809280 bytes 2024-05-17T15:02:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3086795776 bytes 2024-05-17T15:02:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3131862016 bytes 2024-05-17T15:02:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3156122624 bytes 2024-05-17T15:02:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3255412736 bytes 2024-05-17T15:02:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3274379264 bytes 2024-05-17T15:02:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3341601792 bytes 2024-05-17T15:02:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3342206976 bytes 2024-05-17T15:02:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3393516544 bytes 2024-05-17T15:02:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3430722560 bytes 2024-05-17T15:02:56.415 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3449769984 bytes 2024-05-17T15:02:56.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3473736704 bytes 2024-05-17T15:02:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3534456832 bytes 2024-05-17T15:02:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3547837440 bytes 2024-05-17T15:02:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3601451008 bytes 2024-05-17T15:02:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3630781440 bytes 2024-05-17T15:02:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3681413120 bytes 2024-05-17T15:02:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3724311552 bytes 2024-05-17T15:02:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3795393536 bytes 2024-05-17T15:02:57.433 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:02:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:02:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:02:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:02:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-05-17T15:02:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:02:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T15:02:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T15:02:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T15:02:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:02:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:02:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-05-17T15:02:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:02:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T15:02:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local description_pattern= 2024-05-17T15:02:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local service_pattern= 2024-05-17T15:02:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T15:02:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T15:02:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-05-17T15:02:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='test: 2024-05-17T15:02:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: global_id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T15:02:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+replaying 2024-05-17T15:02:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: replaying, {"bytes_per_second":17872213.33,"bytes_per_snapshot":535580160.0,"last_snapshot_bytes":536166400,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715958157,"remote_snapshot_timestamp":1715958157,"replay_state":"idle"} 2024-05-17T15:02:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: service: mirror.2 on smithi195 2024-05-17T15:02:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 15:02:46 2024-05-17T15:02:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: peer_sites: 2024-05-17T15:02:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: name: cluster2 2024-05-17T15:02:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: state: up+stopped 2024-05-17T15:02:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: description: local image is primary 2024-05-17T15:02:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: last_update: 2024-05-17 15:02:47' 2024-05-17T15:02:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ state: .*up+replaying' 2024-05-17T15:02:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:02:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^ description: .*' 2024-05-17T15:02:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":17872213.33,"bytes_per_snapshot":535580160.0,"last_snapshot_bytes":536166400,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715958157,"remote_snapshot_timestamp":1715958157,"replay_state":"idle"} 2024-05-17T15:02:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:02:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T15:02:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep '^up+' 2024-05-17T15:02:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T15:02:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep 'service: *mirror..* on ' 2024-05-17T15:02:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.2 on smithi195 2024-05-17T15:02:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update 2024-05-17T15:02:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-05-17T15:02:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 15:02:46' 2024-05-17T15:02:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-05-17 15:02:46' 2024-05-17T15:02:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:02:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state_pattern=up+replaying 2024-05-17T15:02:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local 'prev_last_update=2024-05-17 15:02:46' 2024-05-17T15:02:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status 2024-05-17T15:02:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ CEPH_ARGS= 2024-05-17T15:02:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-05-17T15:02:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status='OKOKOK142184296mirror.2smithi19519.0.0-3735-gee59d86atrueOK42224307mirror.3smithi19519.0.0-3735-gee59d86afalseOK44024405mirror.0smithi19519.0.0-3735-gee59d86afalseOK45314534mirror.1smithi19519.0.0-3735-gee59d86afalseOKtest72894289-d917-4c92-922e-8f6ade3b96caup+replayingreplaying, {"bytes_per_second":17872213.33,"bytes_per_snapshot":535580160.0,"last_snapshot_bytes":536166400,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1715958157,"remote_snapshot_timestamp":1715958157,"replay_state":"idle"}42184296mirror.2smithi1952024-05-17 15:02:46cluster27e816122-c11e-4a54-9bab-7372f406213bup+stoppedlocal image is primary2024-05-17 15:02:47' 2024-05-17T15:02:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_update state 2024-05-17T15:02:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-05-17T15:02:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ last_update='2024-05-17 15:02:46' 2024-05-17T15:02:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-05-17T15:02:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ state=up+replaying 2024-05-17T15:02:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo up+replaying 2024-05-17T15:02:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep up+replaying 2024-05-17T15:02:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:up+replaying 2024-05-17T15:02:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:02:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:02:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap10 2024-05-17T15:02:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap10 2024-05-17T15:02:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:02:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:02:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:02:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap10 2024-05-17T15:02:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:02:58.852 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7619027355766806) > weight (0.25) 2024-05-17T15:02:58.853 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-05-17T15:02:58.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-05-17T15:02:58.854 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-05-17T15:02:58.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi195.stderr:2024-05-17T15:02:58.853+0000 7fd158fb3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 40593) UID: 1000 2024-05-17T15:03:06.855 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-05-17T15:03:06.855 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:03:06.856 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-05-17T15:03:06.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-05-17T15:03:06.856 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-05-17T15:03:06.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-05-17T15:03:06.860 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2024-05-17T15:03:28.744 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5845309562895964) > weight (0.25) 2024-05-17T15:03:28.744 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-05-17T15:03:28.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-05-17T15:03:28.745 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-05-17T15:03:28.745 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-05-17T15:03:28.745 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:03:28.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:03:28.744+0000 7f2e97b18640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 43094) UID: 1000 2024-05-17T15:03:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-17T15:03:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:03:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-17T15:03:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:03:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:03:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:03:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:03:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:03:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-17T15:03:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:03:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:03:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:03:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:03:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-17T15:03:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:03:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:03:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:03:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:03:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:03:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:03:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:03:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:03:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:03:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:03:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:03:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T15:03:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:03:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:03:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:03:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:03:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:03:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:03:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:03:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:03:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:03:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:03:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:03:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:03:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:03:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:03:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:03:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:03:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:03:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:03:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T15:03:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:03:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:03:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:03:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:03:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T15:03:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:03:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:03:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:03:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:03:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:03:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:03:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:03:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:03:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:03:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:03:29.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:03:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:03:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:03:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:03:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:03:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:03:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:03:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:03:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:03:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:03:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:03:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:03:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:03:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:03:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:03:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:03:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:03:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:03:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:03:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:03:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:03:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:03:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:03:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T15:03:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-05-17T15:03:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:03:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T15:03:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:03:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:03:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:03:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:03:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:03:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:03:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:03:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:03:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:03:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:03:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:03:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:03:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:03:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:03:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:03:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:03:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:03:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:03:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:03:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:03:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:03:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:03:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T15:03:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:03:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:03:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:03:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:03:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:03:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:03:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:03:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:03:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-17T15:03:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:03:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:03:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:03:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:03:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-05-17T15:03:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:03:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:03:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-test.status 2024-05-17T15:03:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-test.status 2024-05-17T15:03:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:03:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T15:03:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-test.status 2024-05-17T15:03:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-test.status 2024-05-17T15:03:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:03:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:03:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-05-17T15:03:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:03:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-05-17T15:03:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 32 2024-05-17T15:03:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T15:03:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:03:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T15:03:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-05-17T15:03:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:03:31.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:03:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=32 2024-05-17T15:03:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:03:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:03:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:03:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:03:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:03:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:03:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:03:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:03:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:03:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:03:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:03:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:03:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:03:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:03:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 32 = '' 2024-05-17T15:03:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:03:32.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:03:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:03:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:03:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:03:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:03:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:03:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:03:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:03:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:03:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:03:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:03:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:03:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 32 = '' 2024-05-17T15:03:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:03:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:03:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:03:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:03:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:03:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:03:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:03:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:03:33.464 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:03:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:03:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:03:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:03:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:03:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 32 = '' 2024-05-17T15:03:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:03:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:03:34.847 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-05-17T15:03:34.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-05-17T15:03:34.847 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-05-17T15:03:34.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-05-17T15:03:34.850 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2024-05-17T15:03:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:03:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:03:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:03:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:03:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:03:35.115 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:03:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=32 2024-05-17T15:03:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 32 = 32 2024-05-17T15:03:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:03:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-05-17T15:03:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:03:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap10 2024-05-17T15:03:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:03:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:03:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:03:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-05-17T15:03:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'test': 2024-05-17T15:03:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 512 MiB in 128 objects 2024-05-17T15:03:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:03:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 20 2024-05-17T15:03:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10bd697942f 2024-05-17T15:03:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10bd697942f 2024-05-17T15:03:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:03:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:03:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:03:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:03:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 14:54:18 2024 2024-05-17T15:03:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:02:43 2024 2024-05-17T15:03:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 14:54:18 2024 2024-05-17T15:03:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:03:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:03:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:03:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T15:03:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:03:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:03:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snap9-clean ']' 2024-05-17T15:03:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap9-clean 2024-05-17T15:03:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap9-clean 2024-05-17T15:03:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:03:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:37.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:03:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap10 2024-05-17T15:03:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap10 2024-05-17T15:03:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:03:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:37.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:03:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ clean_snap_name=snap10-clean 2024-05-17T15:03:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-05-17T15:03:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:03:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap10-clean 2024-05-17T15:03:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-05-17T15:03:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-05-17T15:03:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:03:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-05-17T15:03:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:03:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:03:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:03:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:03:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:03:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror test started 2024-05-17T15:03:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:03:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:03:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:03:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:03:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-05-17T15:03:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:03:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:03:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:03:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:03:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:03:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:03:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:03:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:03:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:03:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:03:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:03:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-05-17T15:03:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:03:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:03:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:03:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:03:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:03:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:03:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:03:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:03:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:03:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:03:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:03:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:03:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:03:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:03:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:03:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:03:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:03:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:03:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:03:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:03:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:03:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:03:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:03:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:03:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:03:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:03:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:03:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:03:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:03:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:03:40.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:03:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:03:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:03:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-05-17T15:03:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:03:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:03:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:03:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:03:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-05-17T15:03:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:03:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:03:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:03:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:03:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:03:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:03:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:03:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:03:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:03:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:03:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:03:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:03:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:03:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:03:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:03:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:03:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:03:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:03:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:03:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:03:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:03:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:03:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:03:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:03:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:03:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:03:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:03:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:03:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:03:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:03:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:03:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:03:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:03:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-05-17T15:03:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-05-17T15:03:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:03:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:03:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:03:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-05-17T15:03:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:03:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:03:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:03:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:03:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:03:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:03:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:03:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:03:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:03:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:03:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:03:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:03:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:03:40.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:03:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:03:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:03:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:03:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:03:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:03:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:03:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:03:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:03:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:03:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:03:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:03:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:03:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:03:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:03:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:03:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:03:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:03:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:03:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:03:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-05-17T15:03:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:03:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:03:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:03:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:03:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:03:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:03:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:03:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:03:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-05-17T15:03:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:03:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:03:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:03:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:03:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-05-17T15:03:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:03:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:03:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-test.status 2024-05-17T15:03:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-test.status 2024-05-17T15:03:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:03:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T15:03:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-test.status 2024-05-17T15:03:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-test.status 2024-05-17T15:03:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:03:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:03:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-05-17T15:03:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:03:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-05-17T15:03:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 34 2024-05-17T15:03:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T15:03:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:03:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.status 2024-05-17T15:03:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-05-17T15:03:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:03:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:03:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=34 2024-05-17T15:03:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:03:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:03:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:03:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:03:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:03:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:03:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:03:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:03:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:03:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:03:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:03:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:03:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:03:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:03:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 34 = '' 2024-05-17T15:03:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:03:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:03:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:03:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:03:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.status 2024-05-17T15:03:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-05-17T15:03:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:03:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:03:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=34 2024-05-17T15:03:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 34 = 34 2024-05-17T15:03:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:03:43.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-05-17T15:03:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:03:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap10-clean 2024-05-17T15:03:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:03:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:03:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:03:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-05-17T15:03:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'test': 2024-05-17T15:03:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 512 MiB in 128 objects 2024-05-17T15:03:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:03:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 21 2024-05-17T15:03:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10bd697942f 2024-05-17T15:03:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10bd697942f 2024-05-17T15:03:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:03:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:03:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:03:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:03:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 14:54:18 2024 2024-05-17T15:03:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:02:43 2024 2024-05-17T15:03:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 14:54:18 2024 2024-05-17T15:03:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:03:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:03:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:03:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 72894289-d917-4c92-922e-8f6ade3b96ca 2024-05-17T15:03:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:03:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:03:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ seq 1 10 2024-05-17T15:03:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:03:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap1 2024-05-17T15:03:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap1 2024-05-17T15:03:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap1 2024-05-17T15:03:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:03:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:03:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap1-clean 2024-05-17T15:03:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap1-clean 2024-05-17T15:03:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap1-clean 2024-05-17T15:03:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:03:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:46.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:03:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:03:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap2 2024-05-17T15:03:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap2 2024-05-17T15:03:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap2 2024-05-17T15:03:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:03:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:03:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap2-clean 2024-05-17T15:03:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap2-clean 2024-05-17T15:03:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap2-clean 2024-05-17T15:03:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:03:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:03:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:03:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap3 2024-05-17T15:03:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap3 2024-05-17T15:03:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap3 2024-05-17T15:03:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:03:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:03:53.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap3-clean 2024-05-17T15:03:53.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap3-clean 2024-05-17T15:03:53.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:53.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:53.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap3-clean 2024-05-17T15:03:53.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:03:53.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:53.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:53.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:03:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:03:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap4 2024-05-17T15:03:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap4 2024-05-17T15:03:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap4 2024-05-17T15:03:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:03:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:56.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:03:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap4-clean 2024-05-17T15:03:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap4-clean 2024-05-17T15:03:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap4-clean 2024-05-17T15:03:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:03:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:58.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:03:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:03:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap5 2024-05-17T15:03:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap5 2024-05-17T15:03:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:03:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:03:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap5 2024-05-17T15:03:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:03:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:03:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:03:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:03:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap5-clean 2024-05-17T15:04:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap5-clean 2024-05-17T15:04:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap5-clean 2024-05-17T15:04:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:04:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:01.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:04:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap6 2024-05-17T15:04:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap6 2024-05-17T15:04:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap6 2024-05-17T15:04:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:04:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap6-clean 2024-05-17T15:04:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap6-clean 2024-05-17T15:04:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap6-clean 2024-05-17T15:04:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:04:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:04.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:04:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap7 2024-05-17T15:04:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap7 2024-05-17T15:04:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap7 2024-05-17T15:04:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:04:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap7-clean 2024-05-17T15:04:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap7-clean 2024-05-17T15:04:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap7-clean 2024-05-17T15:04:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:04:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:04:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap8 2024-05-17T15:04:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap8 2024-05-17T15:04:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap8 2024-05-17T15:04:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:04:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap8-clean 2024-05-17T15:04:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap8-clean 2024-05-17T15:04:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap8-clean 2024-05-17T15:04:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:04:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:12.150 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:12.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:12.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:12.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:04:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap9 2024-05-17T15:04:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap9 2024-05-17T15:04:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap9 2024-05-17T15:04:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:04:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap9-clean 2024-05-17T15:04:14.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap9-clean 2024-05-17T15:04:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap9-clean 2024-05-17T15:04:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:04:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:15.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:04:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap10 2024-05-17T15:04:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap10 2024-05-17T15:04:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap10 2024-05-17T15:04:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:04:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap10-clean 2024-05-17T15:04:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror test snap10-clean 2024-05-17T15:04:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap10-clean 2024-05-17T15:04:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:04:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export 2024-05-17T15:04:18.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:18.229 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-05-17T15:04:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:19.250 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-test.export /tmp/tmp.0i1dljubcH/cluster1-mirror-test.export 2024-05-17T15:04:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ seq 1 10 2024-05-17T15:04:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:04:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap1 2024-05-17T15:04:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-05-17T15:04:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap1 2024-05-17T15:04:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-05-17T15:04:19.861 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-05-17T15:04:19.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-05-17T15:04:19.862 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-05-17T15:04:19.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr:2024-05-17T15:04:19.861+0000 7f4f69ed0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 41295) UID: 1000 2024-05-17T15:04:20.422 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap1-clean 2024-05-17T15:04:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-05-17T15:04:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap1-clean 2024-05-17T15:04:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-05-17T15:04:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:04:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap2 2024-05-17T15:04:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-05-17T15:04:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap2 2024-05-17T15:04:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-05-17T15:04:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap2-clean 2024-05-17T15:04:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-05-17T15:04:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap2-clean 2024-05-17T15:04:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-05-17T15:04:22.862 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-05-17T15:04:22.863 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:04:22.863 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-05-17T15:04:22.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-05-17T15:04:22.863 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-05-17T15:04:22.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-05-17T15:04:22.866 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-05-17T15:04:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:04:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap3 2024-05-17T15:04:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-05-17T15:04:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap3 2024-05-17T15:04:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-05-17T15:04:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap3-clean 2024-05-17T15:04:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-05-17T15:04:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap3-clean 2024-05-17T15:04:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-05-17T15:04:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:04:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap4 2024-05-17T15:04:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-05-17T15:04:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap4 2024-05-17T15:04:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-05-17T15:04:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap4-clean 2024-05-17T15:04:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-05-17T15:04:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap4-clean 2024-05-17T15:04:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-05-17T15:04:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:04:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap5 2024-05-17T15:04:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-05-17T15:04:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap5 2024-05-17T15:04:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-05-17T15:04:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap5-clean 2024-05-17T15:04:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-05-17T15:04:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap5-clean 2024-05-17T15:04:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-05-17T15:04:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:04:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap6 2024-05-17T15:04:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-05-17T15:04:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap6 2024-05-17T15:04:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-05-17T15:04:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap6-clean 2024-05-17T15:04:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-05-17T15:04:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap6-clean 2024-05-17T15:04:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-05-17T15:04:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:04:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap7 2024-05-17T15:04:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-05-17T15:04:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap7 2024-05-17T15:04:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-05-17T15:04:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap7-clean 2024-05-17T15:04:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-05-17T15:04:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap7-clean 2024-05-17T15:04:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-05-17T15:04:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:04:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap8 2024-05-17T15:04:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-05-17T15:04:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap8 2024-05-17T15:04:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-05-17T15:04:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap8-clean 2024-05-17T15:04:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-05-17T15:04:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap8-clean 2024-05-17T15:04:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-05-17T15:04:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:04:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap9 2024-05-17T15:04:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-05-17T15:04:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap9 2024-05-17T15:04:34.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-05-17T15:04:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap9-clean 2024-05-17T15:04:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-05-17T15:04:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap9-clean 2024-05-17T15:04:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-05-17T15:04:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 10` 2024-05-17T15:04:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap10 2024-05-17T15:04:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-05-17T15:04:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap10 2024-05-17T15:04:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-05-17T15:04:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap10-clean 2024-05-17T15:04:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-05-17T15:04:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap=snap10-clean 2024-05-17T15:04:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-05-17T15:04:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing snap: 100% complete...done. 2024-05-17T15:04:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_image_retry cluster2 mirror test 2024-05-17T15:04:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0 1 2 4 8 16 32 2024-05-17T15:04:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0 2024-05-17T15:04:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ remove_image cluster2 mirror test 2024-05-17T15:04:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-05-17T15:04:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-05-17T15:04:38.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi195.stderr:2024-05-17T15:04:38.943+0000 7fb2d2a86640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558315ad7a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-05-17T15:04:38.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi195.stderr:2024-05-17T15:04:38.943+0000 7fb2d8a92640 -1 rbd::mirror::ImageReplayer: 0x558316442f00 [3/72894289-d917-4c92-922e-8f6ade3b96ca] operator(): start failed: (85) Interrupted system call should be restarted 2024-05-17T15:04:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-05-17T15:04:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-05-17T15:04:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:04:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=deleted 2024-05-17T15:04:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image_id= 2024-05-17T15:04:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:04:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n '' 2024-05-17T15:04:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ get_image_id cluster1 mirror test 2024-05-17T15:04:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:04:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local pool=mirror 2024-05-17T15:04:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local image=test 2024-05-17T15:04:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-05-17T15:04:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-17T15:04:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: error opening image test: (2) No such file or directory 2024-05-17T15:04:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image_id= 2024-05-17T15:04:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-05-17T15:04:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.1 2024-05-17T15:04:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-05-17T15:04:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:04:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=test 2024-05-17T15:04:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=deleted 2024-05-17T15:04:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image_id= 2024-05-17T15:04:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=deleted 2024-05-17T15:04:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_image_id 2024-05-17T15:04:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ get_image_id cluster1 mirror test 2024-05-17T15:04:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:04:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local pool=mirror 2024-05-17T15:04:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local image=test 2024-05-17T15:04:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-05-17T15:04:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-05-17T15:04:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: error opening image test: (2) No such file or directory 2024-05-17T15:04:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_image_id= 2024-05-17T15:04:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n '' 2024-05-17T15:04:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test deleted = deleted 2024-05-17T15:04:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ testlog 'TEST: create many images' 2024-05-17T15:04:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ tee -a /tmp/tmp.0i1dljubcH/rbd-mirror.test.log 2024-05-17T15:04:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ date '+%F %T' 2024-05-17T15:04:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17 15:04:41 TEST: create many images 2024-05-17T15:04:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:2024-05-17 15:04:41 TEST: create many images 2024-05-17T15:04:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ snap_name=snap 2024-05-17T15:04:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ seq 1 50 2024-05-17T15:04:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:04:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_1 2024-05-17T15:04:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-05-17T15:04:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:04:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:04:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:04:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_1 128M 2024-05-17T15:04:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:04:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:04:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:04:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:04:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2024-05-17T15:04:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:04:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:04:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-05-17T15:04:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:04:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:04:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-05-17T15:04:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:04:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-05-17T15:04:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_1': 2024-05-17T15:04:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:04:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:04:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:04:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12709d3ba8b4 2024-05-17T15:04:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12709d3ba8b4 2024-05-17T15:04:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:04:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:04:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:04:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:04:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:04:41 2024 2024-05-17T15:04:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:04:41 2024 2024-05-17T15:04:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:04:41 2024 2024-05-17T15:04:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:04:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:04:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: ceaa7752-4708-4453-aad7-8c9fe4b5ad62 2024-05-17T15:04:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_1 2024-05-17T15:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:04:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:04:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=10 2024-05-17T15:04:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:04:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:04:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 22160384 bytes 2024-05-17T15:04:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 43991040 bytes 2024-05-17T15:04:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 76946432 bytes 2024-05-17T15:04:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 101641216 bytes 2024-05-17T15:04:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 186851328 bytes 2024-05-17T15:04:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 222149632 bytes 2024-05-17T15:04:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 292176896 bytes 2024-05-17T15:04:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 326808576 bytes 2024-05-17T15:04:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 379324416 bytes 2024-05-17T15:04:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 440104960 bytes 2024-05-17T15:04:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 502335488 bytes 2024-05-17T15:04:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 537006080 bytes 2024-05-17T15:04:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 600169472 bytes 2024-05-17T15:04:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 676677632 bytes 2024-05-17T15:04:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 750964736 bytes 2024-05-17T15:04:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 834204672 bytes 2024-05-17T15:04:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 844819456 bytes 2024-05-17T15:04:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 865115136 bytes 2024-05-17T15:04:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 984076288 bytes 2024-05-17T15:04:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 1034590208 bytes 2024-05-17T15:04:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 1112695808 bytes 2024-05-17T15:04:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1208323072 bytes 2024-05-17T15:04:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1278926848 bytes 2024-05-17T15:04:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1291265024 bytes 2024-05-17T15:04:45.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1373328384 bytes 2024-05-17T15:04:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1456247808 bytes 2024-05-17T15:04:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1491269632 bytes 2024-05-17T15:04:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1508217856 bytes 2024-05-17T15:04:45.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1561148416 bytes 2024-05-17T15:04:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1567863808 bytes 2024-05-17T15:04:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1576010752 bytes 2024-05-17T15:04:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1613988864 bytes 2024-05-17T15:04:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1687810048 bytes 2024-05-17T15:04:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1726064640 bytes 2024-05-17T15:04:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1773294592 bytes 2024-05-17T15:04:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1798273024 bytes 2024-05-17T15:04:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1821574144 bytes 2024-05-17T15:04:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1901759488 bytes 2024-05-17T15:04:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1923003392 bytes 2024-05-17T15:04:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1982497792 bytes 2024-05-17T15:04:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 2011017216 bytes 2024-05-17T15:04:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 2042809344 bytes 2024-05-17T15:04:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 2169596928 bytes 2024-05-17T15:04:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 2209696768 bytes 2024-05-17T15:04:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2257383424 bytes 2024-05-17T15:04:47.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2287023104 bytes 2024-05-17T15:04:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2345220096 bytes 2024-05-17T15:04:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2365555712 bytes 2024-05-17T15:04:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2416907264 bytes 2024-05-17T15:04:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2461735936 bytes 2024-05-17T15:04:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2520816640 bytes 2024-05-17T15:04:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2607277056 bytes 2024-05-17T15:04:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2672180224 bytes 2024-05-17T15:04:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2672798720 bytes 2024-05-17T15:04:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2733171712 bytes 2024-05-17T15:04:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2739677184 bytes 2024-05-17T15:04:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2791825408 bytes 2024-05-17T15:04:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2800517120 bytes 2024-05-17T15:04:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2865340416 bytes 2024-05-17T15:04:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2887333888 bytes 2024-05-17T15:04:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2929206272 bytes 2024-05-17T15:04:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2969964544 bytes 2024-05-17T15:04:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 3004230656 bytes 2024-05-17T15:04:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 3044729856 bytes 2024-05-17T15:04:49.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 3059565568 bytes 2024-05-17T15:04:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 3079248896 bytes 2024-05-17T15:04:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 3099747328 bytes 2024-05-17T15:04:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 3145126912 bytes 2024-05-17T15:04:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 3244018688 bytes 2024-05-17T15:04:50.471 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 3244618752 bytes 2024-05-17T15:04:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3245210624 bytes 2024-05-17T15:04:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3289797632 bytes 2024-05-17T15:04:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3301554176 bytes 2024-05-17T15:04:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3352159232 bytes 2024-05-17T15:04:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3385041920 bytes 2024-05-17T15:04:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3389920256 bytes 2024-05-17T15:04:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3419264000 bytes 2024-05-17T15:04:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3463662592 bytes 2024-05-17T15:04:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3509331968 bytes 2024-05-17T15:04:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3533088768 bytes 2024-05-17T15:04:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3552659456 bytes 2024-05-17T15:04:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3583074304 bytes 2024-05-17T15:04:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3631669248 bytes 2024-05-17T15:04:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:04:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:04:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:04:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:04:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_2 2024-05-17T15:04:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-05-17T15:04:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:04:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:04:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:04:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_2 128M 2024-05-17T15:04:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:04:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:04:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:04:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:04:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2024-05-17T15:04:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:04:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:04:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-05-17T15:04:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:04:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:04:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-05-17T15:04:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:04:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-05-17T15:04:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_2': 2024-05-17T15:04:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:04:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:04:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:04:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 127bb8f725f7 2024-05-17T15:04:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.127bb8f725f7 2024-05-17T15:04:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:04:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:04:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:04:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:04:51 2024 2024-05-17T15:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:04:51 2024 2024-05-17T15:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:04:51 2024 2024-05-17T15:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 8d44a27c-5bdd-4a78-9b8a-9a4d22d34367 2024-05-17T15:04:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:04:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:04:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_2 2024-05-17T15:04:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:04:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:04:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=7 2024-05-17T15:04:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:04:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:04:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 54514688 bytes 2024-05-17T15:04:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 128544768 bytes 2024-05-17T15:04:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 284326912 bytes 2024-05-17T15:04:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 325964800 bytes 2024-05-17T15:04:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 386979840 bytes 2024-05-17T15:04:53.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 434283520 bytes 2024-05-17T15:04:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 441895936 bytes 2024-05-17T15:04:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 488616960 bytes 2024-05-17T15:04:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 539637760 bytes 2024-05-17T15:04:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 569421824 bytes 2024-05-17T15:04:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 617551872 bytes 2024-05-17T15:04:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 652067840 bytes 2024-05-17T15:04:54.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 720643072 bytes 2024-05-17T15:04:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 785855488 bytes 2024-05-17T15:04:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 812333056 bytes 2024-05-17T15:04:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 853338112 bytes 2024-05-17T15:04:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 876264448 bytes 2024-05-17T15:04:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 883054592 bytes 2024-05-17T15:04:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 930667520 bytes 2024-05-17T15:04:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 944869376 bytes 2024-05-17T15:04:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 1004931072 bytes 2024-05-17T15:04:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1039738880 bytes 2024-05-17T15:04:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1045751808 bytes 2024-05-17T15:04:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1086399488 bytes 2024-05-17T15:04:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1128355840 bytes 2024-05-17T15:04:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1141878784 bytes 2024-05-17T15:04:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1184503808 bytes 2024-05-17T15:04:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1244980224 bytes 2024-05-17T15:04:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1269009408 bytes 2024-05-17T15:04:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1337575424 bytes 2024-05-17T15:04:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1368950784 bytes 2024-05-17T15:04:56.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1406121984 bytes 2024-05-17T15:04:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1406733312 bytes 2024-05-17T15:04:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1500990464 bytes 2024-05-17T15:04:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1540115456 bytes 2024-05-17T15:04:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1560042496 bytes 2024-05-17T15:04:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1622110208 bytes 2024-05-17T15:04:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1655624704 bytes 2024-05-17T15:04:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1752337408 bytes 2024-05-17T15:04:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1752943616 bytes 2024-05-17T15:04:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1785688064 bytes 2024-05-17T15:04:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1797713920 bytes 2024-05-17T15:04:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1823484928 bytes 2024-05-17T15:04:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1883935744 bytes 2024-05-17T15:04:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1922082816 bytes 2024-05-17T15:04:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1961885696 bytes 2024-05-17T15:04:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2020272128 bytes 2024-05-17T15:04:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2110130176 bytes 2024-05-17T15:04:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2187757568 bytes 2024-05-17T15:04:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2217321472 bytes 2024-05-17T15:04:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2240645120 bytes 2024-05-17T15:04:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2269319168 bytes 2024-05-17T15:04:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2319670272 bytes 2024-05-17T15:04:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2381759488 bytes 2024-05-17T15:04:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2469171200 bytes 2024-05-17T15:04:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2516981760 bytes 2024-05-17T15:04:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2557417472 bytes 2024-05-17T15:04:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:04:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:04:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:04:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:04:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_3 2024-05-17T15:04:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-05-17T15:04:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:04:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:04:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:04:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_3 128M 2024-05-17T15:04:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:04:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:04:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:04:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:04:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:04:59.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2024-05-17T15:04:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:04:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:04:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:04:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-05-17T15:04:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:04:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:04:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:04:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:04:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-05-17T15:05:00.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:05:00.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-05-17T15:05:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_3': 2024-05-17T15:05:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:05:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:05:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:05:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 128795991640 2024-05-17T15:05:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.128795991640 2024-05-17T15:05:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:05:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:05:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:05:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:05:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:04:59 2024 2024-05-17T15:05:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:04:59 2024 2024-05-17T15:05:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:04:59 2024 2024-05-17T15:05:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:05:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:05:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 5191746b-1578-4957-a477-41bde526d1e0 2024-05-17T15:05:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:05:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:05:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_3 2024-05-17T15:05:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:05:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:05:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=7 2024-05-17T15:05:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:05:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:05:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 91375616 bytes 2024-05-17T15:05:01.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 92000256 bytes 2024-05-17T15:05:01.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 117277696 bytes 2024-05-17T15:05:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 185571328 bytes 2024-05-17T15:05:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 213269504 bytes 2024-05-17T15:05:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 268308480 bytes 2024-05-17T15:05:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 343610368 bytes 2024-05-17T15:05:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 455162880 bytes 2024-05-17T15:05:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 471448576 bytes 2024-05-17T15:05:02.318 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 495732736 bytes 2024-05-17T15:05:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 547476480 bytes 2024-05-17T15:05:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 562009088 bytes 2024-05-17T15:05:02.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 619634688 bytes 2024-05-17T15:05:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 631205888 bytes 2024-05-17T15:05:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 654901248 bytes 2024-05-17T15:05:03.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 706409472 bytes 2024-05-17T15:05:03.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 794649600 bytes 2024-05-17T15:05:03.372 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 842540032 bytes 2024-05-17T15:05:03.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 894166016 bytes 2024-05-17T15:05:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 919654400 bytes 2024-05-17T15:05:03.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 989998080 bytes 2024-05-17T15:05:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1073895424 bytes 2024-05-17T15:05:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1113962496 bytes 2024-05-17T15:05:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1175791616 bytes 2024-05-17T15:05:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1224627200 bytes 2024-05-17T15:05:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1254011904 bytes 2024-05-17T15:05:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1304392704 bytes 2024-05-17T15:05:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1354004480 bytes 2024-05-17T15:05:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1390784512 bytes 2024-05-17T15:05:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1412977664 bytes 2024-05-17T15:05:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1450522624 bytes 2024-05-17T15:05:05.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1465299968 bytes 2024-05-17T15:05:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1488184320 bytes 2024-05-17T15:05:05.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1550172160 bytes 2024-05-17T15:05:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1670011904 bytes 2024-05-17T15:05:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1690727424 bytes 2024-05-17T15:05:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1738816512 bytes 2024-05-17T15:05:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1769744384 bytes 2024-05-17T15:05:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1797988352 bytes 2024-05-17T15:05:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1801896960 bytes 2024-05-17T15:05:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1842755584 bytes 2024-05-17T15:05:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1918738432 bytes 2024-05-17T15:05:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1958123520 bytes 2024-05-17T15:05:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 2013609984 bytes 2024-05-17T15:05:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2046721024 bytes 2024-05-17T15:05:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2047302656 bytes 2024-05-17T15:05:06.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2093082624 bytes 2024-05-17T15:05:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2134800384 bytes 2024-05-17T15:05:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2141400064 bytes 2024-05-17T15:05:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2183071744 bytes 2024-05-17T15:05:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2194751488 bytes 2024-05-17T15:05:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2215936000 bytes 2024-05-17T15:05:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2233134080 bytes 2024-05-17T15:05:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2243852288 bytes 2024-05-17T15:05:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2257902592 bytes 2024-05-17T15:05:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2261139456 bytes 2024-05-17T15:05:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:05:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:05:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:05:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:05:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:05:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_4 2024-05-17T15:05:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-05-17T15:05:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:05:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:05:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:05:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_4 128M 2024-05-17T15:05:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:05:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:05:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:05:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:05:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2024-05-17T15:05:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:05:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:05:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:05:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-05-17T15:05:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:05:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:05:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-05-17T15:05:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:05:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-05-17T15:05:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_4': 2024-05-17T15:05:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:05:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:05:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:05:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 129338b943b8 2024-05-17T15:05:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.129338b943b8 2024-05-17T15:05:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:05:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:05:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:05:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:05:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:07 2024 2024-05-17T15:05:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:05:07 2024 2024-05-17T15:05:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:07 2024 2024-05-17T15:05:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:05:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:05:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 1fda3480-fa61-4b4b-93a6-2dd063ac50f4 2024-05-17T15:05:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:05:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:05:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_4 2024-05-17T15:05:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:05:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:05:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=6 2024-05-17T15:05:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:05:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:05:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 42826752 bytes 2024-05-17T15:05:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 76553216 bytes 2024-05-17T15:05:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 113944576 bytes 2024-05-17T15:05:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 144899072 bytes 2024-05-17T15:05:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 171909120 bytes 2024-05-17T15:05:09.233 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 254563328 bytes 2024-05-17T15:05:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 305353728 bytes 2024-05-17T15:05:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 363466752 bytes 2024-05-17T15:05:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 391808000 bytes 2024-05-17T15:05:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 418321408 bytes 2024-05-17T15:05:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 464302080 bytes 2024-05-17T15:05:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 477632512 bytes 2024-05-17T15:05:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 557421568 bytes 2024-05-17T15:05:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 585134080 bytes 2024-05-17T15:05:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 636809216 bytes 2024-05-17T15:05:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 720024576 bytes 2024-05-17T15:05:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 823806976 bytes 2024-05-17T15:05:10.802 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 854244352 bytes 2024-05-17T15:05:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 906681344 bytes 2024-05-17T15:05:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 948312064 bytes 2024-05-17T15:05:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 1057383424 bytes 2024-05-17T15:05:11.342 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1063322624 bytes 2024-05-17T15:05:11.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1063926784 bytes 2024-05-17T15:05:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1130227712 bytes 2024-05-17T15:05:11.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1154852864 bytes 2024-05-17T15:05:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1182261248 bytes 2024-05-17T15:05:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1201025024 bytes 2024-05-17T15:05:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1244994560 bytes 2024-05-17T15:05:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1297920000 bytes 2024-05-17T15:05:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1343133696 bytes 2024-05-17T15:05:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1356560384 bytes 2024-05-17T15:05:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1383094272 bytes 2024-05-17T15:05:12.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1427587072 bytes 2024-05-17T15:05:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1435816960 bytes 2024-05-17T15:05:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1449126912 bytes 2024-05-17T15:05:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1490981888 bytes 2024-05-17T15:05:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1517875200 bytes 2024-05-17T15:05:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1598537728 bytes 2024-05-17T15:05:13.217 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1664068608 bytes 2024-05-17T15:05:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1704542208 bytes 2024-05-17T15:05:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1715698688 bytes 2024-05-17T15:05:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1775188992 bytes 2024-05-17T15:05:13.613 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:2024-05-17T15:05:13.610+0000 7f9383ede640 -1 reset not still connected to 0x5569b1342ea0 2024-05-17T15:05:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1832856576 bytes 2024-05-17T15:05:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1861166080 bytes 2024-05-17T15:05:13.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1910350848 bytes 2024-05-17T15:05:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1959484416 bytes 2024-05-17T15:05:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1960100864 bytes 2024-05-17T15:05:14.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 1987651584 bytes 2024-05-17T15:05:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2060381184 bytes 2024-05-17T15:05:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:05:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:05:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:05:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:05:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:05:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_5 2024-05-17T15:05:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-05-17T15:05:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:05:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:05:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:05:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_5 128M 2024-05-17T15:05:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:05:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:05:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:05:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:05:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2024-05-17T15:05:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:05:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:05:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:05:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-05-17T15:05:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:05:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:05:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-05-17T15:05:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:05:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-05-17T15:05:15.495 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_5': 2024-05-17T15:05:15.495 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:05:15.495 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:05:15.495 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:05:15.495 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 129ea1ee3342 2024-05-17T15:05:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.129ea1ee3342 2024-05-17T15:05:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:05:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:05:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:05:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:05:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:14 2024 2024-05-17T15:05:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:05:14 2024 2024-05-17T15:05:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:14 2024 2024-05-17T15:05:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:05:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:05:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 82562a71-d364-4946-b5e9-67440f1071c4 2024-05-17T15:05:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:05:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:05:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_5 2024-05-17T15:05:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:05:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:05:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=9 2024-05-17T15:05:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:05:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:05:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 42446848 bytes 2024-05-17T15:05:15.860 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 139921408 bytes 2024-05-17T15:05:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 191980544 bytes 2024-05-17T15:05:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 246850560 bytes 2024-05-17T15:05:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 247464960 bytes 2024-05-17T15:05:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 263155712 bytes 2024-05-17T15:05:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 279748608 bytes 2024-05-17T15:05:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 290929664 bytes 2024-05-17T15:05:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 303725568 bytes 2024-05-17T15:05:16.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 335016960 bytes 2024-05-17T15:05:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 401690624 bytes 2024-05-17T15:05:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 450747392 bytes 2024-05-17T15:05:17.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 483027968 bytes 2024-05-17T15:05:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 492788736 bytes 2024-05-17T15:05:17.120 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:2024-05-17T15:05:17.117+0000 7f9383ede640 -1 reset not still connected to 0x5569b0a4dc70 2024-05-17T15:05:17.120 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:2024-05-17T15:05:17.117+0000 7f9383ede640 -1 reset not still connected to 0x5569b2db4dd0 2024-05-17T15:05:17.120 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:2024-05-17T15:05:17.118+0000 7f9383ede640 -1 reset not still connected to 0x5569c08e3e10 2024-05-17T15:05:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 498302976 bytes 2024-05-17T15:05:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 512363520 bytes 2024-05-17T15:05:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 512979968 bytes 2024-05-17T15:05:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 594103296 bytes 2024-05-17T15:05:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 633584640 bytes 2024-05-17T15:05:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 672247808 bytes 2024-05-17T15:05:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 691982336 bytes 2024-05-17T15:05:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 747536384 bytes 2024-05-17T15:05:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 790781952 bytes 2024-05-17T15:05:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 821688320 bytes 2024-05-17T15:05:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 929342464 bytes 2024-05-17T15:05:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 987353088 bytes 2024-05-17T15:05:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1039203328 bytes 2024-05-17T15:05:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1046167552 bytes 2024-05-17T15:05:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1113936896 bytes 2024-05-17T15:05:18.851 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.308518351989837) > weight (0.25) 2024-05-17T15:05:18.851 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.29551575601940705) > weight (0.25) 2024-05-17T15:05:18.852 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-05-17T15:05:18.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-05-17T15:05:18.852 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-05-17T15:05:18.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T15:05:18.852+0000 7f575d518640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 38032) UID: 1000 2024-05-17T15:05:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1199863808 bytes 2024-05-17T15:05:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1307505664 bytes 2024-05-17T15:05:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1393347584 bytes 2024-05-17T15:05:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1455572992 bytes 2024-05-17T15:05:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1464100864 bytes 2024-05-17T15:05:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1511847936 bytes 2024-05-17T15:05:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1541332992 bytes 2024-05-17T15:05:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1553188864 bytes 2024-05-17T15:05:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1587528704 bytes 2024-05-17T15:05:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1626719232 bytes 2024-05-17T15:05:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1687231488 bytes 2024-05-17T15:05:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1787634688 bytes 2024-05-17T15:05:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1846336512 bytes 2024-05-17T15:05:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1877602304 bytes 2024-05-17T15:05:20.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1891004416 bytes 2024-05-17T15:05:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1945838592 bytes 2024-05-17T15:05:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1989993472 bytes 2024-05-17T15:05:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1998010368 bytes 2024-05-17T15:05:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2068574208 bytes 2024-05-17T15:05:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2096473088 bytes 2024-05-17T15:05:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2142475264 bytes 2024-05-17T15:05:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2208074752 bytes 2024-05-17T15:05:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2287703040 bytes 2024-05-17T15:05:22.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2318604288 bytes 2024-05-17T15:05:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2348732416 bytes 2024-05-17T15:05:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2408259584 bytes 2024-05-17T15:05:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2428798976 bytes 2024-05-17T15:05:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2470056960 bytes 2024-05-17T15:05:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2535511040 bytes 2024-05-17T15:05:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2587307008 bytes 2024-05-17T15:05:22.853 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-05-17T15:05:22.853 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:05:22.854 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-05-17T15:05:22.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-05-17T15:05:22.854 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-05-17T15:05:22.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-05-17T15:05:22.857 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2024-05-17T15:05:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2637080576 bytes 2024-05-17T15:05:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2674012160 bytes 2024-05-17T15:05:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2735212544 bytes 2024-05-17T15:05:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2773015552 bytes 2024-05-17T15:05:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2822984704 bytes 2024-05-17T15:05:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2841434112 bytes 2024-05-17T15:05:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2861770752 bytes 2024-05-17T15:05:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2918406144 bytes 2024-05-17T15:05:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2951074816 bytes 2024-05-17T15:05:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2979820544 bytes 2024-05-17T15:05:24.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 2987190272 bytes 2024-05-17T15:05:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3037327360 bytes 2024-05-17T15:05:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3127861248 bytes 2024-05-17T15:05:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:05:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:05:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:05:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:05:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:05:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_6 2024-05-17T15:05:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-05-17T15:05:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:05:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:05:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:05:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_6 128M 2024-05-17T15:05:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:05:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:05:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:05:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:05:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2024-05-17T15:05:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:05:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:05:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:05:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-05-17T15:05:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:05:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:05:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-05-17T15:05:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:05:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-05-17T15:05:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_6': 2024-05-17T15:05:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:05:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:05:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:05:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12b0e17fee07 2024-05-17T15:05:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12b0e17fee07 2024-05-17T15:05:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:05:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:05:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:05:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:05:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:24 2024 2024-05-17T15:05:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:05:24 2024 2024-05-17T15:05:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:24 2024 2024-05-17T15:05:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:05:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:05:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: b40480e1-3aef-46bb-a86f-05a1ee0d2d26 2024-05-17T15:05:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:05:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:05:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_6 2024-05-17T15:05:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:05:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:05:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=7 2024-05-17T15:05:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:05:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:05:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 20884480 bytes 2024-05-17T15:05:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 88771584 bytes 2024-05-17T15:05:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 126915584 bytes 2024-05-17T15:05:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 154411008 bytes 2024-05-17T15:05:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 173494272 bytes 2024-05-17T15:05:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 244188160 bytes 2024-05-17T15:05:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 257580032 bytes 2024-05-17T15:05:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 274693120 bytes 2024-05-17T15:05:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 325600256 bytes 2024-05-17T15:05:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 357392384 bytes 2024-05-17T15:05:26.860 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 408838144 bytes 2024-05-17T15:05:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 443499520 bytes 2024-05-17T15:05:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 529765376 bytes 2024-05-17T15:05:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 544634880 bytes 2024-05-17T15:05:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 607450112 bytes 2024-05-17T15:05:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 620979200 bytes 2024-05-17T15:05:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 662154240 bytes 2024-05-17T15:05:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 678790144 bytes 2024-05-17T15:05:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 686849024 bytes 2024-05-17T15:05:27.860 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 704352256 bytes 2024-05-17T15:05:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 730180608 bytes 2024-05-17T15:05:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 743800832 bytes 2024-05-17T15:05:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 765902848 bytes 2024-05-17T15:05:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 806436864 bytes 2024-05-17T15:05:28.433 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 833699840 bytes 2024-05-17T15:05:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 834312192 bytes 2024-05-17T15:05:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 886533120 bytes 2024-05-17T15:05:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 956079104 bytes 2024-05-17T15:05:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 989663232 bytes 2024-05-17T15:05:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1069465600 bytes 2024-05-17T15:05:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1129777152 bytes 2024-05-17T15:05:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1141919744 bytes 2024-05-17T15:05:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1157039104 bytes 2024-05-17T15:05:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1175358464 bytes 2024-05-17T15:05:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1223685120 bytes 2024-05-17T15:05:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1255828480 bytes 2024-05-17T15:05:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1318402048 bytes 2024-05-17T15:05:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1343187968 bytes 2024-05-17T15:05:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1378080768 bytes 2024-05-17T15:05:30.372 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1419339776 bytes 2024-05-17T15:05:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1460657152 bytes 2024-05-17T15:05:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1533282304 bytes 2024-05-17T15:05:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1578505216 bytes 2024-05-17T15:05:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1654631424 bytes 2024-05-17T15:05:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1689822208 bytes 2024-05-17T15:05:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1695489024 bytes 2024-05-17T15:05:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1788197888 bytes 2024-05-17T15:05:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 1804063744 bytes 2024-05-17T15:05:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 1839593472 bytes 2024-05-17T15:05:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 1849551872 bytes 2024-05-17T15:05:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 1935863808 bytes 2024-05-17T15:05:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 1958451200 bytes 2024-05-17T15:05:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2039269376 bytes 2024-05-17T15:05:32.244 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2107300864 bytes 2024-05-17T15:05:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2161848320 bytes 2024-05-17T15:05:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:05:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:05:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:05:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:05:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:05:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_7 2024-05-17T15:05:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-05-17T15:05:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:05:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:05:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:05:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_7 128M 2024-05-17T15:05:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:05:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:05:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:05:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:05:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2024-05-17T15:05:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:05:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:05:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:05:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-05-17T15:05:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:05:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:05:32.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-05-17T15:05:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:05:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-05-17T15:05:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_7': 2024-05-17T15:05:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:05:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:05:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:05:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12bc59353a0b 2024-05-17T15:05:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12bc59353a0b 2024-05-17T15:05:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:05:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:05:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:05:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:05:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:32 2024 2024-05-17T15:05:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:05:32 2024 2024-05-17T15:05:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:32 2024 2024-05-17T15:05:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:05:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:05:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 6e4b54d1-2cee-4ccb-aef6-8e1d0ba0e670 2024-05-17T15:05:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:05:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:05:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_7 2024-05-17T15:05:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:05:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:05:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=14 2024-05-17T15:05:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:05:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:05:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 36641792 bytes 2024-05-17T15:05:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 102147072 bytes 2024-05-17T15:05:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 192575488 bytes 2024-05-17T15:05:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 210611200 bytes 2024-05-17T15:05:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 244688896 bytes 2024-05-17T15:05:34.339 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 255459328 bytes 2024-05-17T15:05:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 293950464 bytes 2024-05-17T15:05:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 352103424 bytes 2024-05-17T15:05:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 390328320 bytes 2024-05-17T15:05:34.831 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 454424576 bytes 2024-05-17T15:05:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 479745024 bytes 2024-05-17T15:05:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 534454272 bytes 2024-05-17T15:05:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 573503488 bytes 2024-05-17T15:05:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 607575040 bytes 2024-05-17T15:05:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 652646400 bytes 2024-05-17T15:05:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 714506240 bytes 2024-05-17T15:05:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 749335552 bytes 2024-05-17T15:05:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 794791936 bytes 2024-05-17T15:05:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 863255552 bytes 2024-05-17T15:05:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 911637504 bytes 2024-05-17T15:05:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 964754432 bytes 2024-05-17T15:05:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 989538304 bytes 2024-05-17T15:05:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1005999104 bytes 2024-05-17T15:05:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1028289536 bytes 2024-05-17T15:05:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1071917056 bytes 2024-05-17T15:05:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1115152384 bytes 2024-05-17T15:05:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1141671936 bytes 2024-05-17T15:05:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1200589824 bytes 2024-05-17T15:05:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1245697024 bytes 2024-05-17T15:05:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1345428480 bytes 2024-05-17T15:05:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1380397056 bytes 2024-05-17T15:05:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1437295616 bytes 2024-05-17T15:05:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1511731200 bytes 2024-05-17T15:05:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1578731520 bytes 2024-05-17T15:05:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1610486784 bytes 2024-05-17T15:05:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1660042240 bytes 2024-05-17T15:05:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1665130496 bytes 2024-05-17T15:05:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1689485312 bytes 2024-05-17T15:05:38.433 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1696820224 bytes 2024-05-17T15:05:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1706520576 bytes 2024-05-17T15:05:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1742597120 bytes 2024-05-17T15:05:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1744450560 bytes 2024-05-17T15:05:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1776978944 bytes 2024-05-17T15:05:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1809866752 bytes 2024-05-17T15:05:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1896673280 bytes 2024-05-17T15:05:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1928934400 bytes 2024-05-17T15:05:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1939512320 bytes 2024-05-17T15:05:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 1978072064 bytes 2024-05-17T15:05:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 1996487680 bytes 2024-05-17T15:05:39.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2024182784 bytes 2024-05-17T15:05:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2061314048 bytes 2024-05-17T15:05:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2087961600 bytes 2024-05-17T15:05:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2111110144 bytes 2024-05-17T15:05:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2193339392 bytes 2024-05-17T15:05:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2207769600 bytes 2024-05-17T15:05:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2230664192 bytes 2024-05-17T15:05:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2267092992 bytes 2024-05-17T15:05:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2367010816 bytes 2024-05-17T15:05:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2386969600 bytes 2024-05-17T15:05:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2387581952 bytes 2024-05-17T15:05:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2398654464 bytes 2024-05-17T15:05:40.867 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.41386832990570355) > weight (0.25) 2024-05-17T15:05:40.868 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6953249003820918) > weight (0.25) 2024-05-17T15:05:40.868 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7981337831458188) > weight (0.25) 2024-05-17T15:05:40.868 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-05-17T15:05:40.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-05-17T15:05:40.868 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-05-17T15:05:40.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi195.stderr:2024-05-17T15:05:40.868+0000 7fb2d9a94640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 38162) UID: 1000 2024-05-17T15:05:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2450393088 bytes 2024-05-17T15:05:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2477249536 bytes 2024-05-17T15:05:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2523880448 bytes 2024-05-17T15:05:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2571504640 bytes 2024-05-17T15:05:41.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2638667776 bytes 2024-05-17T15:05:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2712945664 bytes 2024-05-17T15:05:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2808239104 bytes 2024-05-17T15:05:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2862475264 bytes 2024-05-17T15:05:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 2873016320 bytes 2024-05-17T15:05:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 2930575360 bytes 2024-05-17T15:05:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 2991595520 bytes 2024-05-17T15:05:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3006680064 bytes 2024-05-17T15:05:42.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3072762880 bytes 2024-05-17T15:05:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3157994496 bytes 2024-05-17T15:05:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3178822656 bytes 2024-05-17T15:05:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3241340928 bytes 2024-05-17T15:05:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3241943040 bytes 2024-05-17T15:05:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3256937472 bytes 2024-05-17T15:05:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3276899328 bytes 2024-05-17T15:05:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3316700160 bytes 2024-05-17T15:05:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3333010432 bytes 2024-05-17T15:05:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3368370176 bytes 2024-05-17T15:05:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3466913792 bytes 2024-05-17T15:05:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3481101312 bytes 2024-05-17T15:05:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3581533184 bytes 2024-05-17T15:05:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3661848576 bytes 2024-05-17T15:05:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3719831552 bytes 2024-05-17T15:05:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3769337856 bytes 2024-05-17T15:05:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3820990464 bytes 2024-05-17T15:05:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 3834954752 bytes 2024-05-17T15:05:45.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9200 IOs, 3901557760 bytes 2024-05-17T15:05:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9300 IOs, 3951803392 bytes 2024-05-17T15:05:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9400 IOs, 4042861568 bytes 2024-05-17T15:05:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9500 IOs, 4122815488 bytes 2024-05-17T15:05:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9600 IOs, 4182791168 bytes 2024-05-17T15:05:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9700 IOs, 4253531136 bytes 2024-05-17T15:05:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9800 IOs, 4305185792 bytes 2024-05-17T15:05:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9900 IOs, 4364406784 bytes 2024-05-17T15:05:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10000 IOs, 4469980160 bytes 2024-05-17T15:05:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10100 IOs, 4476602368 bytes 2024-05-17T15:05:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10200 IOs, 4548968448 bytes 2024-05-17T15:05:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10300 IOs, 4582895616 bytes 2024-05-17T15:05:46.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10400 IOs, 4585899008 bytes 2024-05-17T15:05:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10500 IOs, 4591418368 bytes 2024-05-17T15:05:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10600 IOs, 4643167232 bytes 2024-05-17T15:05:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10700 IOs, 4718056448 bytes 2024-05-17T15:05:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10800 IOs, 4752675840 bytes 2024-05-17T15:05:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10900 IOs, 4776208384 bytes 2024-05-17T15:05:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:11000 IOs, 4821232640 bytes 2024-05-17T15:05:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:05:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:05:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:05:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:05:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:05:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_8 2024-05-17T15:05:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-05-17T15:05:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:05:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:05:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:05:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_8 128M 2024-05-17T15:05:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:05:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:05:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:05:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:05:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2024-05-17T15:05:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:05:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:05:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:05:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-05-17T15:05:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:05:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:05:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-05-17T15:05:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:05:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-05-17T15:05:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_8': 2024-05-17T15:05:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:05:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:05:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:05:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12c87da4b989 2024-05-17T15:05:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12c87da4b989 2024-05-17T15:05:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:05:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:05:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:05:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:05:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:47 2024 2024-05-17T15:05:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:05:47 2024 2024-05-17T15:05:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:47 2024 2024-05-17T15:05:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:05:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:05:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 75a0868a-edaf-438e-a7e1-fedf558dccfa 2024-05-17T15:05:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:05:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:05:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_8 2024-05-17T15:05:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:05:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:05:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=8 2024-05-17T15:05:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:05:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:05:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 29948928 bytes 2024-05-17T15:05:48.869 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-05-17T15:05:48.869 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:05:48.870 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-05-17T15:05:48.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-05-17T15:05:48.870 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-05-17T15:05:48.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-05-17T15:05:48.873 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2024-05-17T15:05:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 83787776 bytes 2024-05-17T15:05:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 141301760 bytes 2024-05-17T15:05:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 213253120 bytes 2024-05-17T15:05:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 274242560 bytes 2024-05-17T15:05:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 303442944 bytes 2024-05-17T15:05:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 354523136 bytes 2024-05-17T15:05:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 384114688 bytes 2024-05-17T15:05:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 386774016 bytes 2024-05-17T15:05:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 421171200 bytes 2024-05-17T15:05:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 481858560 bytes 2024-05-17T15:05:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 512486400 bytes 2024-05-17T15:05:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 577544192 bytes 2024-05-17T15:05:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 658276352 bytes 2024-05-17T15:05:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 681136128 bytes 2024-05-17T15:05:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 731724800 bytes 2024-05-17T15:05:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 747156480 bytes 2024-05-17T15:05:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 777116672 bytes 2024-05-17T15:05:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 826011648 bytes 2024-05-17T15:05:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 872197120 bytes 2024-05-17T15:05:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 928783360 bytes 2024-05-17T15:05:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 984254464 bytes 2024-05-17T15:05:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1009779712 bytes 2024-05-17T15:05:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1036249088 bytes 2024-05-17T15:05:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1054229504 bytes 2024-05-17T15:05:51.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1108056064 bytes 2024-05-17T15:05:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1139183616 bytes 2024-05-17T15:05:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1178486784 bytes 2024-05-17T15:05:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1270558720 bytes 2024-05-17T15:05:52.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1347863552 bytes 2024-05-17T15:05:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1375112192 bytes 2024-05-17T15:05:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1437946880 bytes 2024-05-17T15:05:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1464103936 bytes 2024-05-17T15:05:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1603400704 bytes 2024-05-17T15:05:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1613379584 bytes 2024-05-17T15:05:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1685464064 bytes 2024-05-17T15:05:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1699800064 bytes 2024-05-17T15:05:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1739137024 bytes 2024-05-17T15:05:53.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1771462656 bytes 2024-05-17T15:05:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1849013248 bytes 2024-05-17T15:05:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1849625600 bytes 2024-05-17T15:05:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1853558784 bytes 2024-05-17T15:05:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1908499456 bytes 2024-05-17T15:05:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1991195648 bytes 2024-05-17T15:05:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2020415488 bytes 2024-05-17T15:05:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2042545152 bytes 2024-05-17T15:05:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2111472640 bytes 2024-05-17T15:05:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2112226304 bytes 2024-05-17T15:05:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2160075776 bytes 2024-05-17T15:05:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2182523904 bytes 2024-05-17T15:05:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2214493184 bytes 2024-05-17T15:05:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2257484800 bytes 2024-05-17T15:05:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2339996672 bytes 2024-05-17T15:05:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2394425344 bytes 2024-05-17T15:05:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2425470976 bytes 2024-05-17T15:05:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2478675968 bytes 2024-05-17T15:05:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2541175808 bytes 2024-05-17T15:05:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2568280064 bytes 2024-05-17T15:05:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2632391680 bytes 2024-05-17T15:05:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2705566720 bytes 2024-05-17T15:05:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:05:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:05:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:05:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:05:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:05:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_9 2024-05-17T15:05:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-05-17T15:05:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:05:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:05:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:05:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_9 128M 2024-05-17T15:05:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:05:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:05:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:05:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:05:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:05:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2024-05-17T15:05:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:05:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:56.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:05:56.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:05:56.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-05-17T15:05:56.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:56.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:56.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:05:56.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:05:56.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-05-17T15:05:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:05:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-05-17T15:05:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_9': 2024-05-17T15:05:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:05:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:05:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:05:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12dc1b0afdf3 2024-05-17T15:05:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12dc1b0afdf3 2024-05-17T15:05:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:05:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:05:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:05:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:05:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:56 2024 2024-05-17T15:05:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:05:56 2024 2024-05-17T15:05:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:56 2024 2024-05-17T15:05:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:05:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:05:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 6c3aab86-8ad4-4a10-8f48-3d66ec7f5b30 2024-05-17T15:05:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:05:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:05:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_9 2024-05-17T15:05:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:05:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:05:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:05:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:05:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=12 2024-05-17T15:05:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:05:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:05:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 45230080 bytes 2024-05-17T15:05:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 129036288 bytes 2024-05-17T15:05:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 163288064 bytes 2024-05-17T15:05:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 202990592 bytes 2024-05-17T15:05:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 203636736 bytes 2024-05-17T15:05:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 240641024 bytes 2024-05-17T15:05:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 316794880 bytes 2024-05-17T15:05:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 362975232 bytes 2024-05-17T15:05:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 371966976 bytes 2024-05-17T15:05:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 467310592 bytes 2024-05-17T15:05:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 510255104 bytes 2024-05-17T15:05:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 540406784 bytes 2024-05-17T15:05:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 555355136 bytes 2024-05-17T15:05:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 597979136 bytes 2024-05-17T15:05:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 657388544 bytes 2024-05-17T15:05:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 755651584 bytes 2024-05-17T15:05:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 756273152 bytes 2024-05-17T15:05:59.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 795599872 bytes 2024-05-17T15:06:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 810989568 bytes 2024-05-17T15:06:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 828138496 bytes 2024-05-17T15:06:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 850381824 bytes 2024-05-17T15:06:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 897678336 bytes 2024-05-17T15:06:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 965510144 bytes 2024-05-17T15:06:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1010554880 bytes 2024-05-17T15:06:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1026457600 bytes 2024-05-17T15:06:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1101003776 bytes 2024-05-17T15:06:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1122628608 bytes 2024-05-17T15:06:01.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1212022784 bytes 2024-05-17T15:06:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1255772160 bytes 2024-05-17T15:06:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1310388224 bytes 2024-05-17T15:06:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1341357056 bytes 2024-05-17T15:06:01.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1366815744 bytes 2024-05-17T15:06:01.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1432398848 bytes 2024-05-17T15:06:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1474735104 bytes 2024-05-17T15:06:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1587108864 bytes 2024-05-17T15:06:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1605855232 bytes 2024-05-17T15:06:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1684818944 bytes 2024-05-17T15:06:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1720036352 bytes 2024-05-17T15:06:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1766005760 bytes 2024-05-17T15:06:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1802507264 bytes 2024-05-17T15:06:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1847107584 bytes 2024-05-17T15:06:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1933177856 bytes 2024-05-17T15:06:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 2002117632 bytes 2024-05-17T15:06:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 2036089856 bytes 2024-05-17T15:06:03.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2134672384 bytes 2024-05-17T15:06:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2169689088 bytes 2024-05-17T15:06:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2187633664 bytes 2024-05-17T15:06:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2291564544 bytes 2024-05-17T15:06:04.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2292160512 bytes 2024-05-17T15:06:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2327974912 bytes 2024-05-17T15:06:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2360570880 bytes 2024-05-17T15:06:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2433335296 bytes 2024-05-17T15:06:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2457961472 bytes 2024-05-17T15:06:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2507113472 bytes 2024-05-17T15:06:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2553287680 bytes 2024-05-17T15:06:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2626211840 bytes 2024-05-17T15:06:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2663619584 bytes 2024-05-17T15:06:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2674572288 bytes 2024-05-17T15:06:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2678989824 bytes 2024-05-17T15:06:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2679585792 bytes 2024-05-17T15:06:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2753895424 bytes 2024-05-17T15:06:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2754513920 bytes 2024-05-17T15:06:05.820 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2790615040 bytes 2024-05-17T15:06:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2827787264 bytes 2024-05-17T15:06:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2856180736 bytes 2024-05-17T15:06:06.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2856813568 bytes 2024-05-17T15:06:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2930483200 bytes 2024-05-17T15:06:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2971693056 bytes 2024-05-17T15:06:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 3010451456 bytes 2024-05-17T15:06:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 3064117248 bytes 2024-05-17T15:06:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3064712192 bytes 2024-05-17T15:06:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3067412480 bytes 2024-05-17T15:06:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3111741440 bytes 2024-05-17T15:06:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3116466176 bytes 2024-05-17T15:06:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3125045248 bytes 2024-05-17T15:06:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3178708992 bytes 2024-05-17T15:06:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3179322368 bytes 2024-05-17T15:06:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3282225152 bytes 2024-05-17T15:06:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3356993536 bytes 2024-05-17T15:06:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3407095808 bytes 2024-05-17T15:06:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3418287104 bytes 2024-05-17T15:06:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3427630080 bytes 2024-05-17T15:06:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3500511232 bytes 2024-05-17T15:06:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3501124608 bytes 2024-05-17T15:06:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3539402752 bytes 2024-05-17T15:06:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3547124736 bytes 2024-05-17T15:06:08.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3615458304 bytes 2024-05-17T15:06:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3642818560 bytes 2024-05-17T15:06:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3657199616 bytes 2024-05-17T15:06:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3677696000 bytes 2024-05-17T15:06:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 3718415360 bytes 2024-05-17T15:06:09.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9200 IOs, 3755241472 bytes 2024-05-17T15:06:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9300 IOs, 3792036864 bytes 2024-05-17T15:06:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9400 IOs, 3833604096 bytes 2024-05-17T15:06:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9500 IOs, 3891388416 bytes 2024-05-17T15:06:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:06:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:06:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:06:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:06:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:06:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_10 2024-05-17T15:06:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-05-17T15:06:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:06:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:06:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:06:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_10 128M 2024-05-17T15:06:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:06:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:06:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:06:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:06:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2024-05-17T15:06:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:06:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:06:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:06:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-05-17T15:06:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:06:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:06:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-05-17T15:06:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:06:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-05-17T15:06:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_10': 2024-05-17T15:06:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:06:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:06:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:06:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12e458c74e52 2024-05-17T15:06:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12e458c74e52 2024-05-17T15:06:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:06:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:06:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:06:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:06:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:06:09 2024 2024-05-17T15:06:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:06:09 2024 2024-05-17T15:06:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:06:09 2024 2024-05-17T15:06:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:06:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:06:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 95950a54-837a-461f-aafc-36eac3b2a17a 2024-05-17T15:06:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:06:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:06:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_10 2024-05-17T15:06:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:06:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:06:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=5 2024-05-17T15:06:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:06:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:06:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 83963904 bytes 2024-05-17T15:06:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 133272576 bytes 2024-05-17T15:06:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 172745728 bytes 2024-05-17T15:06:10.553 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 273415168 bytes 2024-05-17T15:06:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 314856448 bytes 2024-05-17T15:06:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 328698880 bytes 2024-05-17T15:06:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 354814976 bytes 2024-05-17T15:06:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 419605504 bytes 2024-05-17T15:06:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 455531520 bytes 2024-05-17T15:06:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 490276864 bytes 2024-05-17T15:06:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 538768384 bytes 2024-05-17T15:06:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 563162112 bytes 2024-05-17T15:06:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 609297408 bytes 2024-05-17T15:06:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 671012864 bytes 2024-05-17T15:06:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 723702784 bytes 2024-05-17T15:06:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 752656384 bytes 2024-05-17T15:06:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 799036416 bytes 2024-05-17T15:06:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 864860160 bytes 2024-05-17T15:06:12.433 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 882790400 bytes 2024-05-17T15:06:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 921399296 bytes 2024-05-17T15:06:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 964599808 bytes 2024-05-17T15:06:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 984487936 bytes 2024-05-17T15:06:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1003844608 bytes 2024-05-17T15:06:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1069976576 bytes 2024-05-17T15:06:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1139954688 bytes 2024-05-17T15:06:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1157130240 bytes 2024-05-17T15:06:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1196750848 bytes 2024-05-17T15:06:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1256182784 bytes 2024-05-17T15:06:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1261373440 bytes 2024-05-17T15:06:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1315661824 bytes 2024-05-17T15:06:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1342880768 bytes 2024-05-17T15:06:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1343501312 bytes 2024-05-17T15:06:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1368499200 bytes 2024-05-17T15:06:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1377011712 bytes 2024-05-17T15:06:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1377619968 bytes 2024-05-17T15:06:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1412780032 bytes 2024-05-17T15:06:14.483 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1448536064 bytes 2024-05-17T15:06:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1490921472 bytes 2024-05-17T15:06:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1551511552 bytes 2024-05-17T15:06:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:06:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:06:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:06:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:06:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:06:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_11 2024-05-17T15:06:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-05-17T15:06:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:06:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:06:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:06:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_11 128M 2024-05-17T15:06:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:06:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:06:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:06:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:06:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_11 2024-05-17T15:06:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:06:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:06:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:06:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-05-17T15:06:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:06:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:06:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-05-17T15:06:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:06:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-05-17T15:06:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_11': 2024-05-17T15:06:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:06:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:06:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:06:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12ed63f4c6e2 2024-05-17T15:06:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12ed63f4c6e2 2024-05-17T15:06:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:06:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:06:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:06:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:06:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:06:14 2024 2024-05-17T15:06:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:06:14 2024 2024-05-17T15:06:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:06:14 2024 2024-05-17T15:06:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:06:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:06:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 0e1e48a7-bf0e-42d2-9d09-07eeeeaaf56e 2024-05-17T15:06:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:06:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:06:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_11 2024-05-17T15:06:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:06:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:06:15.860 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=13 2024-05-17T15:06:15.860 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:06:15.860 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:06:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 16792576 bytes 2024-05-17T15:06:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 81159168 bytes 2024-05-17T15:06:16.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 136153088 bytes 2024-05-17T15:06:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 232781824 bytes 2024-05-17T15:06:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 289712128 bytes 2024-05-17T15:06:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 408990720 bytes 2024-05-17T15:06:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 416870400 bytes 2024-05-17T15:06:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 439652352 bytes 2024-05-17T15:06:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 484174848 bytes 2024-05-17T15:06:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 530300928 bytes 2024-05-17T15:06:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 563606528 bytes 2024-05-17T15:06:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 571437056 bytes 2024-05-17T15:06:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 614172672 bytes 2024-05-17T15:06:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 680808448 bytes 2024-05-17T15:06:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 765869056 bytes 2024-05-17T15:06:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 782283776 bytes 2024-05-17T15:06:18.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 804849664 bytes 2024-05-17T15:06:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 806053888 bytes 2024-05-17T15:06:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 881901568 bytes 2024-05-17T15:06:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 944132096 bytes 2024-05-17T15:06:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 1006454784 bytes 2024-05-17T15:06:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1048964096 bytes 2024-05-17T15:06:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1077356544 bytes 2024-05-17T15:06:19.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1175188480 bytes 2024-05-17T15:06:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1228939264 bytes 2024-05-17T15:06:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1246356480 bytes 2024-05-17T15:06:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1263954944 bytes 2024-05-17T15:06:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1302933504 bytes 2024-05-17T15:06:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1365902336 bytes 2024-05-17T15:06:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1390269440 bytes 2024-05-17T15:06:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1438320640 bytes 2024-05-17T15:06:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1464527872 bytes 2024-05-17T15:06:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1474717696 bytes 2024-05-17T15:06:20.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1537072128 bytes 2024-05-17T15:06:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1569399808 bytes 2024-05-17T15:06:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1609952256 bytes 2024-05-17T15:06:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1650375680 bytes 2024-05-17T15:06:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1685814272 bytes 2024-05-17T15:06:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1737271296 bytes 2024-05-17T15:06:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1783685120 bytes 2024-05-17T15:06:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1831417856 bytes 2024-05-17T15:06:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1862721536 bytes 2024-05-17T15:06:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1896771584 bytes 2024-05-17T15:06:21.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1917907968 bytes 2024-05-17T15:06:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1970040832 bytes 2024-05-17T15:06:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1995308032 bytes 2024-05-17T15:06:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2026720256 bytes 2024-05-17T15:06:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2073261056 bytes 2024-05-17T15:06:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2105704448 bytes 2024-05-17T15:06:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2148898816 bytes 2024-05-17T15:06:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2215168000 bytes 2024-05-17T15:06:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2278885376 bytes 2024-05-17T15:06:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2325530624 bytes 2024-05-17T15:06:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2345792512 bytes 2024-05-17T15:06:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2391343104 bytes 2024-05-17T15:06:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2466150400 bytes 2024-05-17T15:06:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2498796544 bytes 2024-05-17T15:06:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2537991168 bytes 2024-05-17T15:06:23.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2555569152 bytes 2024-05-17T15:06:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2569606144 bytes 2024-05-17T15:06:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2607130624 bytes 2024-05-17T15:06:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2646290432 bytes 2024-05-17T15:06:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2655152128 bytes 2024-05-17T15:06:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2747412480 bytes 2024-05-17T15:06:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2790672384 bytes 2024-05-17T15:06:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2851482624 bytes 2024-05-17T15:06:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2910656512 bytes 2024-05-17T15:06:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2978589696 bytes 2024-05-17T15:06:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 3028022272 bytes 2024-05-17T15:06:25.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 3083458560 bytes 2024-05-17T15:06:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3189116928 bytes 2024-05-17T15:06:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3248799744 bytes 2024-05-17T15:06:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3294731264 bytes 2024-05-17T15:06:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3325750272 bytes 2024-05-17T15:06:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3330547712 bytes 2024-05-17T15:06:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3341570048 bytes 2024-05-17T15:06:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3353074688 bytes 2024-05-17T15:06:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3385523200 bytes 2024-05-17T15:06:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3441303552 bytes 2024-05-17T15:06:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3477148672 bytes 2024-05-17T15:06:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3516476416 bytes 2024-05-17T15:06:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3565533184 bytes 2024-05-17T15:06:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3588044800 bytes 2024-05-17T15:06:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3612895232 bytes 2024-05-17T15:06:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3637956608 bytes 2024-05-17T15:06:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3671631872 bytes 2024-05-17T15:06:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3734694912 bytes 2024-05-17T15:06:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3779076096 bytes 2024-05-17T15:06:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3802903552 bytes 2024-05-17T15:06:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3805724672 bytes 2024-05-17T15:06:27.877 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 3849765888 bytes 2024-05-17T15:06:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9200 IOs, 3854756864 bytes 2024-05-17T15:06:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9300 IOs, 3892967424 bytes 2024-05-17T15:06:28.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9400 IOs, 3930873856 bytes 2024-05-17T15:06:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9500 IOs, 3943933952 bytes 2024-05-17T15:06:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9600 IOs, 3988902912 bytes 2024-05-17T15:06:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9700 IOs, 4009851904 bytes 2024-05-17T15:06:28.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9800 IOs, 4074822656 bytes 2024-05-17T15:06:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9900 IOs, 4096909312 bytes 2024-05-17T15:06:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:06:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:06:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:06:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:06:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:06:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_12 2024-05-17T15:06:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-05-17T15:06:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_12 2024-05-17T15:06:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:06:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:06:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_12 128M 2024-05-17T15:06:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_12 2024-05-17T15:06:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:06:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:06:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:06:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_12 2024-05-17T15:06:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:06:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:06:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:06:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-05-17T15:06:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_12 2024-05-17T15:06:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:06:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-05-17T15:06:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:06:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-05-17T15:06:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_12': 2024-05-17T15:06:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:06:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:06:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:06:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12f6b692e5e8 2024-05-17T15:06:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12f6b692e5e8 2024-05-17T15:06:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:06:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:06:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:06:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:06:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:06:28 2024 2024-05-17T15:06:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:06:28 2024 2024-05-17T15:06:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:06:28 2024 2024-05-17T15:06:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:06:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:06:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: c50f08e5-ef2e-4cc9-afd7-fb18dd3f03dc 2024-05-17T15:06:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:06:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:06:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_12 2024-05-17T15:06:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_12 2024-05-17T15:06:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:06:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=6 2024-05-17T15:06:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:06:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:06:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 15558656 bytes 2024-05-17T15:06:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 26621952 bytes 2024-05-17T15:06:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 69621760 bytes 2024-05-17T15:06:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 82710528 bytes 2024-05-17T15:06:30.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 133371904 bytes 2024-05-17T15:06:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 157182976 bytes 2024-05-17T15:06:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 201579520 bytes 2024-05-17T15:06:30.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 264102912 bytes 2024-05-17T15:06:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 300060672 bytes 2024-05-17T15:06:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 338190336 bytes 2024-05-17T15:06:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 393775104 bytes 2024-05-17T15:06:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 433913856 bytes 2024-05-17T15:06:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 471511040 bytes 2024-05-17T15:06:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 488676352 bytes 2024-05-17T15:06:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 546246656 bytes 2024-05-17T15:06:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 581505024 bytes 2024-05-17T15:06:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 643102720 bytes 2024-05-17T15:06:31.831 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 753947648 bytes 2024-05-17T15:06:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 824966144 bytes 2024-05-17T15:06:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 912526336 bytes 2024-05-17T15:06:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 999383040 bytes 2024-05-17T15:06:32.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1052070912 bytes 2024-05-17T15:06:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1150620672 bytes 2024-05-17T15:06:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1172050944 bytes 2024-05-17T15:06:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1210664960 bytes 2024-05-17T15:06:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1255346176 bytes 2024-05-17T15:06:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1281132544 bytes 2024-05-17T15:06:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1297363968 bytes 2024-05-17T15:06:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1315438592 bytes 2024-05-17T15:06:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1381173248 bytes 2024-05-17T15:06:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1437638656 bytes 2024-05-17T15:06:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1507243008 bytes 2024-05-17T15:06:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1507863552 bytes 2024-05-17T15:06:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1552737280 bytes 2024-05-17T15:06:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1583646720 bytes 2024-05-17T15:06:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1584823296 bytes 2024-05-17T15:06:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1620919296 bytes 2024-05-17T15:06:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1663272960 bytes 2024-05-17T15:06:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1679630336 bytes 2024-05-17T15:06:34.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1743320064 bytes 2024-05-17T15:06:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1818577920 bytes 2024-05-17T15:06:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1870375936 bytes 2024-05-17T15:06:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1913763840 bytes 2024-05-17T15:06:35.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1956476928 bytes 2024-05-17T15:06:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:06:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:06:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:06:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:06:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:06:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_13 2024-05-17T15:06:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-05-17T15:06:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_13 2024-05-17T15:06:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:06:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:06:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_13 128M 2024-05-17T15:06:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_13 2024-05-17T15:06:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:06:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:06:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:06:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_13 2024-05-17T15:06:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:06:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:06:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:06:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-05-17T15:06:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_13 2024-05-17T15:06:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:06:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-05-17T15:06:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:06:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-05-17T15:06:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_13': 2024-05-17T15:06:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:06:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:06:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:06:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12fe220ceac0 2024-05-17T15:06:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12fe220ceac0 2024-05-17T15:06:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:06:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:06:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:06:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:06:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:06:35 2024 2024-05-17T15:06:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:06:35 2024 2024-05-17T15:06:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:06:35 2024 2024-05-17T15:06:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:06:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:06:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 76d62c45-9c33-4b38-aeb9-d12dc0c2247a 2024-05-17T15:06:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:06:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:06:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_13 2024-05-17T15:06:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_13 2024-05-17T15:06:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:06:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=10 2024-05-17T15:06:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:06:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:06:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 43177984 bytes 2024-05-17T15:06:36.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 59246592 bytes 2024-05-17T15:06:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 122193920 bytes 2024-05-17T15:06:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 159538176 bytes 2024-05-17T15:06:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 204475392 bytes 2024-05-17T15:06:37.317 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 210063360 bytes 2024-05-17T15:06:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 296301568 bytes 2024-05-17T15:06:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 370930688 bytes 2024-05-17T15:06:37.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 403495936 bytes 2024-05-17T15:06:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 468377600 bytes 2024-05-17T15:06:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 542351360 bytes 2024-05-17T15:06:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 559552512 bytes 2024-05-17T15:06:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 658964480 bytes 2024-05-17T15:06:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 659563520 bytes 2024-05-17T15:06:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 703621120 bytes 2024-05-17T15:06:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 773239808 bytes 2024-05-17T15:06:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 799158272 bytes 2024-05-17T15:06:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 838811648 bytes 2024-05-17T15:06:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 900218880 bytes 2024-05-17T15:06:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 985542656 bytes 2024-05-17T15:06:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 1024941056 bytes 2024-05-17T15:06:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1032036352 bytes 2024-05-17T15:06:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1158152192 bytes 2024-05-17T15:06:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1190893568 bytes 2024-05-17T15:06:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1212475392 bytes 2024-05-17T15:06:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1222625280 bytes 2024-05-17T15:06:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1260553216 bytes 2024-05-17T15:06:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1342031872 bytes 2024-05-17T15:06:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1370793984 bytes 2024-05-17T15:06:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1403373568 bytes 2024-05-17T15:06:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1419491328 bytes 2024-05-17T15:06:40.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1456069632 bytes 2024-05-17T15:06:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1460679680 bytes 2024-05-17T15:06:41.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1484472320 bytes 2024-05-17T15:06:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1507139584 bytes 2024-05-17T15:06:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1557293056 bytes 2024-05-17T15:06:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1587111936 bytes 2024-05-17T15:06:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1608537088 bytes 2024-05-17T15:06:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1625908224 bytes 2024-05-17T15:06:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1718180864 bytes 2024-05-17T15:06:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1780064256 bytes 2024-05-17T15:06:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1780698112 bytes 2024-05-17T15:06:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1845482496 bytes 2024-05-17T15:06:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1973637120 bytes 2024-05-17T15:06:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2003275776 bytes 2024-05-17T15:06:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2036310016 bytes 2024-05-17T15:06:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2039714816 bytes 2024-05-17T15:06:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2087459840 bytes 2024-05-17T15:06:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2088096768 bytes 2024-05-17T15:06:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2111774720 bytes 2024-05-17T15:06:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2135450624 bytes 2024-05-17T15:06:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2141254656 bytes 2024-05-17T15:06:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2220942336 bytes 2024-05-17T15:06:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2240988160 bytes 2024-05-17T15:06:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2260615168 bytes 2024-05-17T15:06:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2305784832 bytes 2024-05-17T15:06:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2351179776 bytes 2024-05-17T15:06:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2437863424 bytes 2024-05-17T15:06:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2502134784 bytes 2024-05-17T15:06:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2548397056 bytes 2024-05-17T15:06:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2602510336 bytes 2024-05-17T15:06:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2618981376 bytes 2024-05-17T15:06:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2624595968 bytes 2024-05-17T15:06:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2660782080 bytes 2024-05-17T15:06:45.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2697947136 bytes 2024-05-17T15:06:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2769200128 bytes 2024-05-17T15:06:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2817357824 bytes 2024-05-17T15:06:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2826323968 bytes 2024-05-17T15:06:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2837948416 bytes 2024-05-17T15:06:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 2852948992 bytes 2024-05-17T15:06:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 2856492032 bytes 2024-05-17T15:06:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 2863922176 bytes 2024-05-17T15:06:45.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 2888467456 bytes 2024-05-17T15:06:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 2889089024 bytes 2024-05-17T15:06:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 2958759936 bytes 2024-05-17T15:06:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 2969168896 bytes 2024-05-17T15:06:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 2986759168 bytes 2024-05-17T15:06:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3031334912 bytes 2024-05-17T15:06:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:06:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:06:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:06:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:06:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:06:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_14 2024-05-17T15:06:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-05-17T15:06:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_14 2024-05-17T15:06:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:06:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:06:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_14 128M 2024-05-17T15:06:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_14 2024-05-17T15:06:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:06:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:06:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:06:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_14 2024-05-17T15:06:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:06:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:06:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:06:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-05-17T15:06:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_14 2024-05-17T15:06:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:06:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-05-17T15:06:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:06:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-05-17T15:06:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_14': 2024-05-17T15:06:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:06:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:06:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:06:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 13093f518870 2024-05-17T15:06:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.13093f518870 2024-05-17T15:06:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:06:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:06:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:06:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:06:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:06:46 2024 2024-05-17T15:06:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:06:46 2024 2024-05-17T15:06:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:06:46 2024 2024-05-17T15:06:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:06:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:06:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 9e41e805-f938-4777-a0dd-ce32b1acfcf1 2024-05-17T15:06:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:06:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:06:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_14 2024-05-17T15:06:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_14 2024-05-17T15:06:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:06:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=8 2024-05-17T15:06:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:06:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:06:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 34202624 bytes 2024-05-17T15:06:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 85696512 bytes 2024-05-17T15:06:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 147038208 bytes 2024-05-17T15:06:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 147658752 bytes 2024-05-17T15:06:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 188238848 bytes 2024-05-17T15:06:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 198906880 bytes 2024-05-17T15:06:48.430 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 229245952 bytes 2024-05-17T15:06:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 249593856 bytes 2024-05-17T15:06:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 295014400 bytes 2024-05-17T15:06:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 301217792 bytes 2024-05-17T15:06:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 324958208 bytes 2024-05-17T15:06:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 373194752 bytes 2024-05-17T15:06:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 403688448 bytes 2024-05-17T15:06:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 468956160 bytes 2024-05-17T15:06:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 532528128 bytes 2024-05-17T15:06:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 579396608 bytes 2024-05-17T15:06:49.692 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 620988416 bytes 2024-05-17T15:06:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 671021056 bytes 2024-05-17T15:06:49.859 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9052813758792803) > weight (0.25) 2024-05-17T15:06:49.859 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-05-17T15:06:49.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-05-17T15:06:49.859 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-05-17T15:06:49.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:06:49.858+0000 7f243dc2a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 44805) UID: 1000 2024-05-17T15:06:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 693798912 bytes 2024-05-17T15:06:50.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 707076096 bytes 2024-05-17T15:06:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 710556672 bytes 2024-05-17T15:06:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 713147392 bytes 2024-05-17T15:06:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 771373056 bytes 2024-05-17T15:06:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 816791552 bytes 2024-05-17T15:06:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 876503040 bytes 2024-05-17T15:06:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 890321920 bytes 2024-05-17T15:06:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 935303168 bytes 2024-05-17T15:06:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 983083008 bytes 2024-05-17T15:06:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1110757376 bytes 2024-05-17T15:06:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1168656384 bytes 2024-05-17T15:06:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1196938240 bytes 2024-05-17T15:06:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1231625216 bytes 2024-05-17T15:06:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1300134912 bytes 2024-05-17T15:06:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1354175488 bytes 2024-05-17T15:06:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1354806272 bytes 2024-05-17T15:06:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1396737024 bytes 2024-05-17T15:06:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1414268928 bytes 2024-05-17T15:06:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1470058496 bytes 2024-05-17T15:06:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1561075712 bytes 2024-05-17T15:06:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1597471744 bytes 2024-05-17T15:06:53.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1669548032 bytes 2024-05-17T15:06:53.160 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1716866048 bytes 2024-05-17T15:06:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1779965952 bytes 2024-05-17T15:06:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1783021568 bytes 2024-05-17T15:06:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1845991424 bytes 2024-05-17T15:06:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1882703872 bytes 2024-05-17T15:06:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1883305984 bytes 2024-05-17T15:06:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 1899624448 bytes 2024-05-17T15:06:54.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 1963938816 bytes 2024-05-17T15:06:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2001469440 bytes 2024-05-17T15:06:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2040967168 bytes 2024-05-17T15:06:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2041586688 bytes 2024-05-17T15:06:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2076463104 bytes 2024-05-17T15:06:54.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2116531200 bytes 2024-05-17T15:06:54.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2133816320 bytes 2024-05-17T15:06:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2189878272 bytes 2024-05-17T15:06:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2256320512 bytes 2024-05-17T15:06:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2292962304 bytes 2024-05-17T15:06:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2307140608 bytes 2024-05-17T15:06:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2338429952 bytes 2024-05-17T15:06:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2380070912 bytes 2024-05-17T15:06:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:06:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:06:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:06:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:06:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:06:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_15 2024-05-17T15:06:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-05-17T15:06:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:06:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:06:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:06:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:55.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_15 128M 2024-05-17T15:06:55.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:55.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:55.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:55.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:55.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:06:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:06:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:06:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:06:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:06:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_15 2024-05-17T15:06:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:06:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:06:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:06:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-05-17T15:06:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:06:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:06:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-05-17T15:06:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:06:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-05-17T15:06:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_15': 2024-05-17T15:06:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:06:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:06:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:06:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1312d1640c82 2024-05-17T15:06:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1312d1640c82 2024-05-17T15:06:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:06:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:06:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:06:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:06:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:06:55 2024 2024-05-17T15:06:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:06:55 2024 2024-05-17T15:06:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:06:55 2024 2024-05-17T15:06:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:06:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:06:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: fb0ce489-3f67-44e1-971c-03b1005074a7 2024-05-17T15:06:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:06:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:06:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_15 2024-05-17T15:06:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:06:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:06:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:06:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:06:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=11 2024-05-17T15:06:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:06:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:06:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 22443008 bytes 2024-05-17T15:06:56.860 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-05-17T15:06:56.861 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:06:56.861 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-05-17T15:06:56.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-05-17T15:06:56.862 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-05-17T15:06:56.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-05-17T15:06:56.865 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2024-05-17T15:06:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 75347968 bytes 2024-05-17T15:06:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 152927232 bytes 2024-05-17T15:06:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 184104960 bytes 2024-05-17T15:06:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 237554688 bytes 2024-05-17T15:06:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 238752768 bytes 2024-05-17T15:06:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 284391424 bytes 2024-05-17T15:06:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 396744704 bytes 2024-05-17T15:06:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 408619008 bytes 2024-05-17T15:06:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 433596416 bytes 2024-05-17T15:06:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 453835776 bytes 2024-05-17T15:06:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 494422016 bytes 2024-05-17T15:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 575050752 bytes 2024-05-17T15:06:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 681041920 bytes 2024-05-17T15:06:58.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 727143424 bytes 2024-05-17T15:06:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 774712320 bytes 2024-05-17T15:06:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 791170048 bytes 2024-05-17T15:06:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 842351616 bytes 2024-05-17T15:06:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 884038656 bytes 2024-05-17T15:06:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 935517184 bytes 2024-05-17T15:06:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 991517696 bytes 2024-05-17T15:06:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 993720320 bytes 2024-05-17T15:06:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1026097152 bytes 2024-05-17T15:06:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1064892416 bytes 2024-05-17T15:06:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1105446912 bytes 2024-05-17T15:07:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1144984576 bytes 2024-05-17T15:07:00.151 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1157292032 bytes 2024-05-17T15:07:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1216712704 bytes 2024-05-17T15:07:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1268870144 bytes 2024-05-17T15:07:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1277495296 bytes 2024-05-17T15:07:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1311114240 bytes 2024-05-17T15:07:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1359811584 bytes 2024-05-17T15:07:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1360432128 bytes 2024-05-17T15:07:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1530858496 bytes 2024-05-17T15:07:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1572334592 bytes 2024-05-17T15:07:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1618806784 bytes 2024-05-17T15:07:01.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1626594304 bytes 2024-05-17T15:07:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1691372544 bytes 2024-05-17T15:07:01.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1731371008 bytes 2024-05-17T15:07:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1796415488 bytes 2024-05-17T15:07:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1850787840 bytes 2024-05-17T15:07:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1898977280 bytes 2024-05-17T15:07:02.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1946144768 bytes 2024-05-17T15:07:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1972119552 bytes 2024-05-17T15:07:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2026217472 bytes 2024-05-17T15:07:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2080507904 bytes 2024-05-17T15:07:02.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2137109504 bytes 2024-05-17T15:07:03.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2183260160 bytes 2024-05-17T15:07:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2262426624 bytes 2024-05-17T15:07:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2279109632 bytes 2024-05-17T15:07:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2321514496 bytes 2024-05-17T15:07:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2414321664 bytes 2024-05-17T15:07:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2509347840 bytes 2024-05-17T15:07:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2552199168 bytes 2024-05-17T15:07:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2600701952 bytes 2024-05-17T15:07:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2705701888 bytes 2024-05-17T15:07:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2795566080 bytes 2024-05-17T15:07:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2839780352 bytes 2024-05-17T15:07:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2905102336 bytes 2024-05-17T15:07:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2927608832 bytes 2024-05-17T15:07:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2985140224 bytes 2024-05-17T15:07:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 3005867008 bytes 2024-05-17T15:07:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 3007478784 bytes 2024-05-17T15:07:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 3008119808 bytes 2024-05-17T15:07:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 3038771200 bytes 2024-05-17T15:07:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 3094033408 bytes 2024-05-17T15:07:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 3162377216 bytes 2024-05-17T15:07:05.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 3223240704 bytes 2024-05-17T15:07:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 3247497216 bytes 2024-05-17T15:07:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 3322827776 bytes 2024-05-17T15:07:06.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3348311040 bytes 2024-05-17T15:07:06.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3369815040 bytes 2024-05-17T15:07:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3441952768 bytes 2024-05-17T15:07:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3487524864 bytes 2024-05-17T15:07:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3514045440 bytes 2024-05-17T15:07:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3595594752 bytes 2024-05-17T15:07:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3650523136 bytes 2024-05-17T15:07:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3671981056 bytes 2024-05-17T15:07:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3761838080 bytes 2024-05-17T15:07:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:07:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:07:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:07:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:07:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:07:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_16 2024-05-17T15:07:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-05-17T15:07:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:07:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:07:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:07:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_16 128M 2024-05-17T15:07:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:07:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:07:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:07:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:07:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_16 2024-05-17T15:07:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:07:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:07:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:07:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-05-17T15:07:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:07:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:07:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-05-17T15:07:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:07:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-05-17T15:07:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_16': 2024-05-17T15:07:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:07:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:07:08.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1322855b6665 2024-05-17T15:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1322855b6665 2024-05-17T15:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:07:07 2024 2024-05-17T15:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:07:07 2024 2024-05-17T15:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:07:07 2024 2024-05-17T15:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 9cc94201-e324-4319-a646-d571d0e3df05 2024-05-17T15:07:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:07:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:07:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_16 2024-05-17T15:07:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:07:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:07:08.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=12 2024-05-17T15:07:08.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:07:08.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:07:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 52136960 bytes 2024-05-17T15:07:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 52758528 bytes 2024-05-17T15:07:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 53365760 bytes 2024-05-17T15:07:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 96538624 bytes 2024-05-17T15:07:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 147795968 bytes 2024-05-17T15:07:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 187350016 bytes 2024-05-17T15:07:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 192744448 bytes 2024-05-17T15:07:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 232950784 bytes 2024-05-17T15:07:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 278648832 bytes 2024-05-17T15:07:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 350547968 bytes 2024-05-17T15:07:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 431489024 bytes 2024-05-17T15:07:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 432115712 bytes 2024-05-17T15:07:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 432736256 bytes 2024-05-17T15:07:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 467730432 bytes 2024-05-17T15:07:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 493036544 bytes 2024-05-17T15:07:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 500189184 bytes 2024-05-17T15:07:10.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 511331328 bytes 2024-05-17T15:07:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 534222848 bytes 2024-05-17T15:07:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 555252736 bytes 2024-05-17T15:07:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 581472256 bytes 2024-05-17T15:07:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 677364736 bytes 2024-05-17T15:07:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 708827136 bytes 2024-05-17T15:07:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 747270144 bytes 2024-05-17T15:07:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 747878400 bytes 2024-05-17T15:07:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 773090304 bytes 2024-05-17T15:07:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 804281344 bytes 2024-05-17T15:07:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 848846848 bytes 2024-05-17T15:07:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 896748544 bytes 2024-05-17T15:07:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 938383360 bytes 2024-05-17T15:07:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 946164736 bytes 2024-05-17T15:07:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 955369472 bytes 2024-05-17T15:07:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1013340160 bytes 2024-05-17T15:07:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1063828480 bytes 2024-05-17T15:07:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1082835968 bytes 2024-05-17T15:07:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1188933632 bytes 2024-05-17T15:07:13.433 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1255109632 bytes 2024-05-17T15:07:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1332429824 bytes 2024-05-17T15:07:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1372401664 bytes 2024-05-17T15:07:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1454986240 bytes 2024-05-17T15:07:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1486286848 bytes 2024-05-17T15:07:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1531678720 bytes 2024-05-17T15:07:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1549829120 bytes 2024-05-17T15:07:14.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1607283712 bytes 2024-05-17T15:07:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1636057088 bytes 2024-05-17T15:07:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1713385472 bytes 2024-05-17T15:07:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1749093376 bytes 2024-05-17T15:07:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1753018368 bytes 2024-05-17T15:07:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 1771347968 bytes 2024-05-17T15:07:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 1822099456 bytes 2024-05-17T15:07:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 1859340288 bytes 2024-05-17T15:07:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 1908122624 bytes 2024-05-17T15:07:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 1972666368 bytes 2024-05-17T15:07:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2030284800 bytes 2024-05-17T15:07:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2047724544 bytes 2024-05-17T15:07:15.874 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.48689433944964033) > weight (0.25) 2024-05-17T15:07:15.874 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9184423046247331) > weight (0.25) 2024-05-17T15:07:15.874 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7910226382292668) > weight (0.25) 2024-05-17T15:07:15.874 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-05-17T15:07:15.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-05-17T15:07:15.875 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-05-17T15:07:15.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi195.stderr:2024-05-17T15:07:15.874+0000 7fceabb9d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 47517) UID: 1000 2024-05-17T15:07:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2061566976 bytes 2024-05-17T15:07:16.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2083892224 bytes 2024-05-17T15:07:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2129460224 bytes 2024-05-17T15:07:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2166578176 bytes 2024-05-17T15:07:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2245996544 bytes 2024-05-17T15:07:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2250526720 bytes 2024-05-17T15:07:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2260170752 bytes 2024-05-17T15:07:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2276371456 bytes 2024-05-17T15:07:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2336770048 bytes 2024-05-17T15:07:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2350903296 bytes 2024-05-17T15:07:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2351504384 bytes 2024-05-17T15:07:17.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2392930304 bytes 2024-05-17T15:07:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2506032128 bytes 2024-05-17T15:07:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2591299584 bytes 2024-05-17T15:07:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2616146944 bytes 2024-05-17T15:07:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 2670292992 bytes 2024-05-17T15:07:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 2699786240 bytes 2024-05-17T15:07:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 2720373760 bytes 2024-05-17T15:07:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 2739873792 bytes 2024-05-17T15:07:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 2836847616 bytes 2024-05-17T15:07:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 2837466112 bytes 2024-05-17T15:07:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 2879094784 bytes 2024-05-17T15:07:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 2889894912 bytes 2024-05-17T15:07:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 2890523648 bytes 2024-05-17T15:07:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 2922182656 bytes 2024-05-17T15:07:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 2963019776 bytes 2024-05-17T15:07:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3019513856 bytes 2024-05-17T15:07:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3035938816 bytes 2024-05-17T15:07:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3057108992 bytes 2024-05-17T15:07:19.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3106534400 bytes 2024-05-17T15:07:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3155365888 bytes 2024-05-17T15:07:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3186014208 bytes 2024-05-17T15:07:19.876 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-05-17T15:07:19.876 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:07:19.877 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-05-17T15:07:19.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-05-17T15:07:19.877 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-05-17T15:07:19.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-05-17T15:07:19.880 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2024-05-17T15:07:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3241662464 bytes 2024-05-17T15:07:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3329081344 bytes 2024-05-17T15:07:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3350544384 bytes 2024-05-17T15:07:20.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3374668800 bytes 2024-05-17T15:07:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 3405355008 bytes 2024-05-17T15:07:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_17 2024-05-17T15:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-05-17T15:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:07:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:07:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_17 128M 2024-05-17T15:07:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:07:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:07:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:07:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:07:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_17 2024-05-17T15:07:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:07:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:07:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:07:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-05-17T15:07:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:07:20.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:07:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-05-17T15:07:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:07:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-05-17T15:07:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_17': 2024-05-17T15:07:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:07:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:07:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:07:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 133543856571 2024-05-17T15:07:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.133543856571 2024-05-17T15:07:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:07:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:07:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:07:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:07:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:07:20 2024 2024-05-17T15:07:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:07:20 2024 2024-05-17T15:07:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:07:20 2024 2024-05-17T15:07:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:07:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:07:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: bef15777-4cd7-47fb-8829-12dabc805b90 2024-05-17T15:07:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:07:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:07:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_17 2024-05-17T15:07:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:07:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:07:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=14 2024-05-17T15:07:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:07:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:07:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 578560 bytes 2024-05-17T15:07:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 19507200 bytes 2024-05-17T15:07:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 56173568 bytes 2024-05-17T15:07:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 139549696 bytes 2024-05-17T15:07:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 167437312 bytes 2024-05-17T15:07:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 250660864 bytes 2024-05-17T15:07:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 285523968 bytes 2024-05-17T15:07:22.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 349208576 bytes 2024-05-17T15:07:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 378457088 bytes 2024-05-17T15:07:23.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 425098240 bytes 2024-05-17T15:07:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 425701376 bytes 2024-05-17T15:07:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 429357056 bytes 2024-05-17T15:07:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 501437440 bytes 2024-05-17T15:07:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 554582016 bytes 2024-05-17T15:07:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 560989184 bytes 2024-05-17T15:07:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 596517888 bytes 2024-05-17T15:07:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 644383744 bytes 2024-05-17T15:07:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 657491968 bytes 2024-05-17T15:07:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 727564288 bytes 2024-05-17T15:07:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 740518912 bytes 2024-05-17T15:07:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 763083776 bytes 2024-05-17T15:07:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 791533568 bytes 2024-05-17T15:07:24.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 847008768 bytes 2024-05-17T15:07:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 897822720 bytes 2024-05-17T15:07:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 906850304 bytes 2024-05-17T15:07:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 950210560 bytes 2024-05-17T15:07:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1009762304 bytes 2024-05-17T15:07:25.323 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1087960064 bytes 2024-05-17T15:07:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1113012224 bytes 2024-05-17T15:07:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1184484352 bytes 2024-05-17T15:07:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1223243776 bytes 2024-05-17T15:07:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1228621824 bytes 2024-05-17T15:07:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1257274368 bytes 2024-05-17T15:07:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1281832960 bytes 2024-05-17T15:07:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1381127168 bytes 2024-05-17T15:07:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1411873792 bytes 2024-05-17T15:07:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1520125952 bytes 2024-05-17T15:07:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1520740352 bytes 2024-05-17T15:07:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1568830464 bytes 2024-05-17T15:07:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1650124800 bytes 2024-05-17T15:07:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1658760192 bytes 2024-05-17T15:07:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1707492352 bytes 2024-05-17T15:07:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1732908032 bytes 2024-05-17T15:07:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1779997696 bytes 2024-05-17T15:07:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1851751424 bytes 2024-05-17T15:07:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1895463936 bytes 2024-05-17T15:07:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1947341824 bytes 2024-05-17T15:07:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 1975552000 bytes 2024-05-17T15:07:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2022800384 bytes 2024-05-17T15:07:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2108019712 bytes 2024-05-17T15:07:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2108638208 bytes 2024-05-17T15:07:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2139915264 bytes 2024-05-17T15:07:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2151224320 bytes 2024-05-17T15:07:28.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2169140224 bytes 2024-05-17T15:07:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2234150912 bytes 2024-05-17T15:07:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2283285504 bytes 2024-05-17T15:07:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2305055744 bytes 2024-05-17T15:07:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2341945344 bytes 2024-05-17T15:07:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2342569984 bytes 2024-05-17T15:07:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2364067840 bytes 2024-05-17T15:07:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2385021952 bytes 2024-05-17T15:07:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2443902976 bytes 2024-05-17T15:07:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2508647424 bytes 2024-05-17T15:07:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2604747776 bytes 2024-05-17T15:07:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2642636800 bytes 2024-05-17T15:07:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2671107072 bytes 2024-05-17T15:07:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2734505984 bytes 2024-05-17T15:07:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2788001792 bytes 2024-05-17T15:07:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2812577792 bytes 2024-05-17T15:07:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 2836021248 bytes 2024-05-17T15:07:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 2851742720 bytes 2024-05-17T15:07:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 2913469440 bytes 2024-05-17T15:07:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 2919922688 bytes 2024-05-17T15:07:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3003825152 bytes 2024-05-17T15:07:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3113311232 bytes 2024-05-17T15:07:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3175479296 bytes 2024-05-17T15:07:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3200684032 bytes 2024-05-17T15:07:32.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3308887040 bytes 2024-05-17T15:07:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3321779200 bytes 2024-05-17T15:07:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3376616448 bytes 2024-05-17T15:07:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3429812224 bytes 2024-05-17T15:07:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3509220352 bytes 2024-05-17T15:07:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3510137856 bytes 2024-05-17T15:07:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3548071936 bytes 2024-05-17T15:07:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3566918656 bytes 2024-05-17T15:07:33.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3633823744 bytes 2024-05-17T15:07:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3694820352 bytes 2024-05-17T15:07:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3772446720 bytes 2024-05-17T15:07:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3806321664 bytes 2024-05-17T15:07:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3837108224 bytes 2024-05-17T15:07:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 3870922752 bytes 2024-05-17T15:07:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9200 IOs, 3903648768 bytes 2024-05-17T15:07:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9300 IOs, 3949973504 bytes 2024-05-17T15:07:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9400 IOs, 4015729664 bytes 2024-05-17T15:07:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9500 IOs, 4016337920 bytes 2024-05-17T15:07:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9600 IOs, 4029454336 bytes 2024-05-17T15:07:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9700 IOs, 4104796160 bytes 2024-05-17T15:07:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9800 IOs, 4149192704 bytes 2024-05-17T15:07:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9900 IOs, 4238631936 bytes 2024-05-17T15:07:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10000 IOs, 4265723904 bytes 2024-05-17T15:07:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10100 IOs, 4310546432 bytes 2024-05-17T15:07:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10200 IOs, 4386917376 bytes 2024-05-17T15:07:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10300 IOs, 4445308928 bytes 2024-05-17T15:07:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:07:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:07:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:07:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:07:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:07:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_18 2024-05-17T15:07:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-05-17T15:07:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:07:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:07:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:07:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_18 128M 2024-05-17T15:07:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:07:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:07:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:07:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:07:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_18 2024-05-17T15:07:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:07:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:07:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:07:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-05-17T15:07:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:07:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:07:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-05-17T15:07:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:07:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-05-17T15:07:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_18': 2024-05-17T15:07:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:07:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:07:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:07:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 133dc02aba29 2024-05-17T15:07:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.133dc02aba29 2024-05-17T15:07:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:07:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:07:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:07:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:07:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:07:35 2024 2024-05-17T15:07:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:07:35 2024 2024-05-17T15:07:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:07:35 2024 2024-05-17T15:07:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:07:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:07:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: e3330572-d02a-4104-a14f-d83973929a2c 2024-05-17T15:07:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:07:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:07:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_18 2024-05-17T15:07:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:07:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:07:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=11 2024-05-17T15:07:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:07:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:07:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 3943424 bytes 2024-05-17T15:07:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 78790656 bytes 2024-05-17T15:07:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 147139584 bytes 2024-05-17T15:07:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 162803712 bytes 2024-05-17T15:07:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 205461504 bytes 2024-05-17T15:07:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 234864640 bytes 2024-05-17T15:07:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 256775168 bytes 2024-05-17T15:07:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 271480832 bytes 2024-05-17T15:07:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 349121536 bytes 2024-05-17T15:07:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 349742080 bytes 2024-05-17T15:07:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 452813824 bytes 2024-05-17T15:07:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 486338560 bytes 2024-05-17T15:07:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 499262464 bytes 2024-05-17T15:07:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 517031936 bytes 2024-05-17T15:07:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 572092416 bytes 2024-05-17T15:07:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 630161408 bytes 2024-05-17T15:07:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 679121920 bytes 2024-05-17T15:07:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 772268032 bytes 2024-05-17T15:07:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 819087360 bytes 2024-05-17T15:07:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 853703680 bytes 2024-05-17T15:07:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 861603840 bytes 2024-05-17T15:07:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 986612736 bytes 2024-05-17T15:07:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 990245888 bytes 2024-05-17T15:07:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1122149376 bytes 2024-05-17T15:07:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1146104832 bytes 2024-05-17T15:07:40.420 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1196861440 bytes 2024-05-17T15:07:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1307257856 bytes 2024-05-17T15:07:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1312515072 bytes 2024-05-17T15:07:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1317026816 bytes 2024-05-17T15:07:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1396227072 bytes 2024-05-17T15:07:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1455628288 bytes 2024-05-17T15:07:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1493721088 bytes 2024-05-17T15:07:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1537239040 bytes 2024-05-17T15:07:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1571551232 bytes 2024-05-17T15:07:41.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1628329984 bytes 2024-05-17T15:07:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1630014464 bytes 2024-05-17T15:07:41.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1653688320 bytes 2024-05-17T15:07:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1695337472 bytes 2024-05-17T15:07:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1728399360 bytes 2024-05-17T15:07:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1846268928 bytes 2024-05-17T15:07:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1903554560 bytes 2024-05-17T15:07:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1975692288 bytes 2024-05-17T15:07:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1998147584 bytes 2024-05-17T15:07:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 2030190592 bytes 2024-05-17T15:07:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2069853184 bytes 2024-05-17T15:07:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2117659648 bytes 2024-05-17T15:07:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2141557760 bytes 2024-05-17T15:07:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2195217408 bytes 2024-05-17T15:07:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2242933760 bytes 2024-05-17T15:07:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2287723520 bytes 2024-05-17T15:07:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2310094848 bytes 2024-05-17T15:07:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2428934144 bytes 2024-05-17T15:07:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2489252864 bytes 2024-05-17T15:07:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2569080832 bytes 2024-05-17T15:07:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2611174400 bytes 2024-05-17T15:07:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2668252160 bytes 2024-05-17T15:07:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2767492096 bytes 2024-05-17T15:07:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2821765120 bytes 2024-05-17T15:07:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2907835392 bytes 2024-05-17T15:07:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2922309632 bytes 2024-05-17T15:07:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2952237056 bytes 2024-05-17T15:07:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2987877376 bytes 2024-05-17T15:07:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 3026180096 bytes 2024-05-17T15:07:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 3116668928 bytes 2024-05-17T15:07:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 3181990912 bytes 2024-05-17T15:07:46.259 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 3264605184 bytes 2024-05-17T15:07:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 3288987648 bytes 2024-05-17T15:07:46.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 3373995008 bytes 2024-05-17T15:07:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 3378040832 bytes 2024-05-17T15:07:46.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 3380551680 bytes 2024-05-17T15:07:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3407843328 bytes 2024-05-17T15:07:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3476891648 bytes 2024-05-17T15:07:47.235 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3567389696 bytes 2024-05-17T15:07:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3623960576 bytes 2024-05-17T15:07:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3669476352 bytes 2024-05-17T15:07:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3689964544 bytes 2024-05-17T15:07:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:07:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:07:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:07:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:07:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:07:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_19 2024-05-17T15:07:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-05-17T15:07:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_19 2024-05-17T15:07:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:07:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:07:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_19 128M 2024-05-17T15:07:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_19 2024-05-17T15:07:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:07:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:07:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:07:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_19 2024-05-17T15:07:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:07:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:07:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:07:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-05-17T15:07:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_19 2024-05-17T15:07:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:07:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-05-17T15:07:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:07:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-05-17T15:07:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_19': 2024-05-17T15:07:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:07:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:07:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:07:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 13478ae737f2 2024-05-17T15:07:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.13478ae737f2 2024-05-17T15:07:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:07:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:07:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:07:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:07:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:07:47 2024 2024-05-17T15:07:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:07:47 2024 2024-05-17T15:07:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:07:47 2024 2024-05-17T15:07:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:07:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:07:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 3df98478-a489-4a33-b01c-132fa8241534 2024-05-17T15:07:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:07:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:07:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_19 2024-05-17T15:07:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_19 2024-05-17T15:07:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:07:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=10 2024-05-17T15:07:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:07:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:07:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 41597952 bytes 2024-05-17T15:07:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 81277952 bytes 2024-05-17T15:07:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 163230720 bytes 2024-05-17T15:07:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 209405952 bytes 2024-05-17T15:07:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 213320704 bytes 2024-05-17T15:07:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 264724480 bytes 2024-05-17T15:07:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 289953792 bytes 2024-05-17T15:07:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 352153600 bytes 2024-05-17T15:07:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 400565248 bytes 2024-05-17T15:07:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 450475008 bytes 2024-05-17T15:07:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 487106560 bytes 2024-05-17T15:07:50.358 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 509094912 bytes 2024-05-17T15:07:50.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 531643392 bytes 2024-05-17T15:07:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 554601472 bytes 2024-05-17T15:07:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 572947456 bytes 2024-05-17T15:07:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 599159808 bytes 2024-05-17T15:07:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 630589440 bytes 2024-05-17T15:07:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 638349312 bytes 2024-05-17T15:07:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 694833152 bytes 2024-05-17T15:07:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 746700800 bytes 2024-05-17T15:07:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 794436608 bytes 2024-05-17T15:07:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 818310144 bytes 2024-05-17T15:07:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 906685440 bytes 2024-05-17T15:07:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 961965056 bytes 2024-05-17T15:07:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1011668992 bytes 2024-05-17T15:07:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1073001472 bytes 2024-05-17T15:07:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1092614144 bytes 2024-05-17T15:07:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1113023488 bytes 2024-05-17T15:07:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1124790272 bytes 2024-05-17T15:07:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1206653952 bytes 2024-05-17T15:07:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1242267648 bytes 2024-05-17T15:07:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1301012480 bytes 2024-05-17T15:07:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1366449152 bytes 2024-05-17T15:07:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1409395712 bytes 2024-05-17T15:07:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1428260864 bytes 2024-05-17T15:07:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1490060288 bytes 2024-05-17T15:07:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1534960640 bytes 2024-05-17T15:07:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1583866880 bytes 2024-05-17T15:07:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1602493440 bytes 2024-05-17T15:07:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1644945408 bytes 2024-05-17T15:07:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1666072576 bytes 2024-05-17T15:07:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1691637760 bytes 2024-05-17T15:07:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1744666624 bytes 2024-05-17T15:07:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1759313920 bytes 2024-05-17T15:07:54.522 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:2024-05-17T15:07:54.521+0000 7f9383ede640 -1 reset not still connected to 0x5569b62924e0 2024-05-17T15:07:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1783870464 bytes 2024-05-17T15:07:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1854051328 bytes 2024-05-17T15:07:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1856414720 bytes 2024-05-17T15:07:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 1936917504 bytes 2024-05-17T15:07:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2005198848 bytes 2024-05-17T15:07:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2025921536 bytes 2024-05-17T15:07:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2072866816 bytes 2024-05-17T15:07:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2104458240 bytes 2024-05-17T15:07:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2105657344 bytes 2024-05-17T15:07:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2164196352 bytes 2024-05-17T15:07:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2226758656 bytes 2024-05-17T15:07:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2246808576 bytes 2024-05-17T15:07:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2333919232 bytes 2024-05-17T15:07:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2378233856 bytes 2024-05-17T15:07:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2378856448 bytes 2024-05-17T15:07:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2436965376 bytes 2024-05-17T15:07:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2456318976 bytes 2024-05-17T15:07:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2536561664 bytes 2024-05-17T15:07:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2616706048 bytes 2024-05-17T15:07:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2619950080 bytes 2024-05-17T15:07:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2670008320 bytes 2024-05-17T15:07:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2699416576 bytes 2024-05-17T15:07:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2727001088 bytes 2024-05-17T15:07:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2839933952 bytes 2024-05-17T15:07:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2913034240 bytes 2024-05-17T15:07:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 2960525312 bytes 2024-05-17T15:07:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 2995084288 bytes 2024-05-17T15:07:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3072841728 bytes 2024-05-17T15:07:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3089522688 bytes 2024-05-17T15:07:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3100221440 bytes 2024-05-17T15:07:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:07:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:07:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:07:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:07:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:07:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_20 2024-05-17T15:07:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-05-17T15:07:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_20 2024-05-17T15:07:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:07:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:07:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_20 128M 2024-05-17T15:07:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_20 2024-05-17T15:07:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:07:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:07:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:07:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:07:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_20 2024-05-17T15:07:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:07:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:07:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:07:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-05-17T15:07:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_20 2024-05-17T15:07:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:07:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-05-17T15:07:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:07:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-05-17T15:07:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_20': 2024-05-17T15:07:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:07:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:07:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:07:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1350d5f1a1c3 2024-05-17T15:07:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1350d5f1a1c3 2024-05-17T15:07:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:07:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:07:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:07:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:07:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:07:58 2024 2024-05-17T15:07:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:07:58 2024 2024-05-17T15:07:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:07:58 2024 2024-05-17T15:07:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:07:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:07:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 99995afa-10de-4385-a379-455ec2be8a34 2024-05-17T15:07:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:07:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:07:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_20 2024-05-17T15:07:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:07:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:07:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_20 2024-05-17T15:07:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:07:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=7 2024-05-17T15:07:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:07:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:07:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 24584192 bytes 2024-05-17T15:08:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 37144576 bytes 2024-05-17T15:08:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 48825344 bytes 2024-05-17T15:08:00.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 105715712 bytes 2024-05-17T15:08:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 204712960 bytes 2024-05-17T15:08:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 270380032 bytes 2024-05-17T15:08:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 317341696 bytes 2024-05-17T15:08:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 350493696 bytes 2024-05-17T15:08:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 378498048 bytes 2024-05-17T15:08:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 424287232 bytes 2024-05-17T15:08:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 470939648 bytes 2024-05-17T15:08:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 570975232 bytes 2024-05-17T15:08:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 620985344 bytes 2024-05-17T15:08:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 649615360 bytes 2024-05-17T15:08:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 700361728 bytes 2024-05-17T15:08:02.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 700956672 bytes 2024-05-17T15:08:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 723662848 bytes 2024-05-17T15:08:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 754920448 bytes 2024-05-17T15:08:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 787683328 bytes 2024-05-17T15:08:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 809964544 bytes 2024-05-17T15:08:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 893878272 bytes 2024-05-17T15:08:02.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 950329344 bytes 2024-05-17T15:08:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1027387392 bytes 2024-05-17T15:08:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1079663616 bytes 2024-05-17T15:08:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1110534144 bytes 2024-05-17T15:08:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1224042496 bytes 2024-05-17T15:08:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1274647552 bytes 2024-05-17T15:08:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1316158464 bytes 2024-05-17T15:08:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1393953792 bytes 2024-05-17T15:08:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1426349056 bytes 2024-05-17T15:08:04.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1434141696 bytes 2024-05-17T15:08:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1485894656 bytes 2024-05-17T15:08:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1533264896 bytes 2024-05-17T15:08:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1560071168 bytes 2024-05-17T15:08:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1576611840 bytes 2024-05-17T15:08:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1592152064 bytes 2024-05-17T15:08:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1635215360 bytes 2024-05-17T15:08:05.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1648647168 bytes 2024-05-17T15:08:05.120 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1675853824 bytes 2024-05-17T15:08:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1677756416 bytes 2024-05-17T15:08:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1719526400 bytes 2024-05-17T15:08:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1730970624 bytes 2024-05-17T15:08:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1764087808 bytes 2024-05-17T15:08:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1865854976 bytes 2024-05-17T15:08:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1866465280 bytes 2024-05-17T15:08:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1933895680 bytes 2024-05-17T15:08:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2018317312 bytes 2024-05-17T15:08:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2036163584 bytes 2024-05-17T15:08:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2080390144 bytes 2024-05-17T15:08:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2083459072 bytes 2024-05-17T15:08:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2120621056 bytes 2024-05-17T15:08:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:08:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:08:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:08:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:08:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:08:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_21 2024-05-17T15:08:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-05-17T15:08:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_21 2024-05-17T15:08:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:08:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:08:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_21 128M 2024-05-17T15:08:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_21 2024-05-17T15:08:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:08:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:08:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:08:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_21 2024-05-17T15:08:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:08:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:08:06.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:08:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-05-17T15:08:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_21 2024-05-17T15:08:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:08:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-05-17T15:08:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:08:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-05-17T15:08:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_21': 2024-05-17T15:08:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:08:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:08:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:08:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1358615b6c5c 2024-05-17T15:08:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1358615b6c5c 2024-05-17T15:08:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:08:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:08:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:08:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:08:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:08:06 2024 2024-05-17T15:08:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:08:06 2024 2024-05-17T15:08:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:08:06 2024 2024-05-17T15:08:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:08:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:08:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 7fede8a9-d664-48e9-b43a-740b49fc17be 2024-05-17T15:08:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:08:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:08:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_21 2024-05-17T15:08:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_21 2024-05-17T15:08:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:08:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=13 2024-05-17T15:08:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:08:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:08:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 16834560 bytes 2024-05-17T15:08:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 49673216 bytes 2024-05-17T15:08:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 92099584 bytes 2024-05-17T15:08:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 93541376 bytes 2024-05-17T15:08:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 113604608 bytes 2024-05-17T15:08:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 138664960 bytes 2024-05-17T15:08:08.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 148789248 bytes 2024-05-17T15:08:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 197293056 bytes 2024-05-17T15:08:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 205686784 bytes 2024-05-17T15:08:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 238211072 bytes 2024-05-17T15:08:09.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 326544384 bytes 2024-05-17T15:08:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 343671808 bytes 2024-05-17T15:08:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 391005184 bytes 2024-05-17T15:08:09.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 443083776 bytes 2024-05-17T15:08:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 490466304 bytes 2024-05-17T15:08:09.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 530286592 bytes 2024-05-17T15:08:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 548503552 bytes 2024-05-17T15:08:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 585183232 bytes 2024-05-17T15:08:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 668140544 bytes 2024-05-17T15:08:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 668743680 bytes 2024-05-17T15:08:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 741329920 bytes 2024-05-17T15:08:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 823571456 bytes 2024-05-17T15:08:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 863315968 bytes 2024-05-17T15:08:10.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 927255552 bytes 2024-05-17T15:08:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 971097088 bytes 2024-05-17T15:08:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1051568128 bytes 2024-05-17T15:08:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1073740800 bytes 2024-05-17T15:08:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1074382848 bytes 2024-05-17T15:08:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1151769600 bytes 2024-05-17T15:08:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1209910272 bytes 2024-05-17T15:08:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1237209088 bytes 2024-05-17T15:08:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1353692160 bytes 2024-05-17T15:08:12.317 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1385523200 bytes 2024-05-17T15:08:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1447109632 bytes 2024-05-17T15:08:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1472501760 bytes 2024-05-17T15:08:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1517155328 bytes 2024-05-17T15:08:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1530827776 bytes 2024-05-17T15:08:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1548719104 bytes 2024-05-17T15:08:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1592027136 bytes 2024-05-17T15:08:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1626120192 bytes 2024-05-17T15:08:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1649182720 bytes 2024-05-17T15:08:13.444 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1705971712 bytes 2024-05-17T15:08:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1770717184 bytes 2024-05-17T15:08:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1788947456 bytes 2024-05-17T15:08:13.805 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1810171904 bytes 2024-05-17T15:08:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1867739136 bytes 2024-05-17T15:08:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1903560704 bytes 2024-05-17T15:08:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 1936473088 bytes 2024-05-17T15:08:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2013431808 bytes 2024-05-17T15:08:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2089006080 bytes 2024-05-17T15:08:14.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2113070080 bytes 2024-05-17T15:08:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2135008256 bytes 2024-05-17T15:08:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2157471744 bytes 2024-05-17T15:08:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2197675008 bytes 2024-05-17T15:08:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2227153920 bytes 2024-05-17T15:08:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2270124032 bytes 2024-05-17T15:08:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2278769664 bytes 2024-05-17T15:08:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2279377920 bytes 2024-05-17T15:08:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2279995392 bytes 2024-05-17T15:08:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2326382592 bytes 2024-05-17T15:08:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2372504576 bytes 2024-05-17T15:08:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2445967360 bytes 2024-05-17T15:08:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2481518592 bytes 2024-05-17T15:08:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2525683712 bytes 2024-05-17T15:08:16.378 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2555297792 bytes 2024-05-17T15:08:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2612662272 bytes 2024-05-17T15:08:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2653981696 bytes 2024-05-17T15:08:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2701751296 bytes 2024-05-17T15:08:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2741200896 bytes 2024-05-17T15:08:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 2761722880 bytes 2024-05-17T15:08:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 2804710400 bytes 2024-05-17T15:08:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 2865506304 bytes 2024-05-17T15:08:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 2933952512 bytes 2024-05-17T15:08:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 2955012096 bytes 2024-05-17T15:08:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3010136064 bytes 2024-05-17T15:08:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3034909696 bytes 2024-05-17T15:08:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3064925184 bytes 2024-05-17T15:08:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3136877568 bytes 2024-05-17T15:08:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3250064384 bytes 2024-05-17T15:08:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3268666368 bytes 2024-05-17T15:08:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3292243968 bytes 2024-05-17T15:08:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3310302208 bytes 2024-05-17T15:08:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3359285248 bytes 2024-05-17T15:08:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3453158400 bytes 2024-05-17T15:08:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3517573120 bytes 2024-05-17T15:08:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3581538304 bytes 2024-05-17T15:08:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3626801152 bytes 2024-05-17T15:08:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3683600384 bytes 2024-05-17T15:08:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3735328768 bytes 2024-05-17T15:08:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3805791232 bytes 2024-05-17T15:08:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 3850423296 bytes 2024-05-17T15:08:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9200 IOs, 3891488768 bytes 2024-05-17T15:08:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9300 IOs, 3927026688 bytes 2024-05-17T15:08:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9400 IOs, 3959459840 bytes 2024-05-17T15:08:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9500 IOs, 3986689024 bytes 2024-05-17T15:08:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:08:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:08:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:08:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:08:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:08:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_22 2024-05-17T15:08:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-05-17T15:08:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:08:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:08:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:08:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_22 128M 2024-05-17T15:08:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:08:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:08:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:08:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:08:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_22 2024-05-17T15:08:20.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:08:20.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:20.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:08:20.877 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:08:20.877 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-05-17T15:08:20.877 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:20.877 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:20.877 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:08:20.877 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:08:20.877 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-05-17T15:08:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:08:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-05-17T15:08:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_22': 2024-05-17T15:08:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:08:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:08:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:08:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1360fb18428f 2024-05-17T15:08:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1360fb18428f 2024-05-17T15:08:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:08:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:08:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:08:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:08:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:08:20 2024 2024-05-17T15:08:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:08:20 2024 2024-05-17T15:08:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:08:20 2024 2024-05-17T15:08:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:08:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:08:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 7b475e11-a35f-4917-9f29-282df00e1098 2024-05-17T15:08:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:08:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:08:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_22 2024-05-17T15:08:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:08:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:08:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=9 2024-05-17T15:08:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:08:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:08:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 38163456 bytes 2024-05-17T15:08:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 62946304 bytes 2024-05-17T15:08:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 100527104 bytes 2024-05-17T15:08:22.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 123296768 bytes 2024-05-17T15:08:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 189552640 bytes 2024-05-17T15:08:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 192846848 bytes 2024-05-17T15:08:22.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 205488128 bytes 2024-05-17T15:08:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 239892480 bytes 2024-05-17T15:08:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 285720576 bytes 2024-05-17T15:08:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 310425600 bytes 2024-05-17T15:08:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 354726912 bytes 2024-05-17T15:08:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 390098944 bytes 2024-05-17T15:08:23.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 411244544 bytes 2024-05-17T15:08:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 448569344 bytes 2024-05-17T15:08:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 509450240 bytes 2024-05-17T15:08:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 559830016 bytes 2024-05-17T15:08:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 586900480 bytes 2024-05-17T15:08:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 647619584 bytes 2024-05-17T15:08:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 660660224 bytes 2024-05-17T15:08:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 723509248 bytes 2024-05-17T15:08:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 776030208 bytes 2024-05-17T15:08:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 795883520 bytes 2024-05-17T15:08:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 868794368 bytes 2024-05-17T15:08:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 896347136 bytes 2024-05-17T15:08:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 943582208 bytes 2024-05-17T15:08:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1005523968 bytes 2024-05-17T15:08:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1060882432 bytes 2024-05-17T15:08:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1122830336 bytes 2024-05-17T15:08:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1175006208 bytes 2024-05-17T15:08:26.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1232246784 bytes 2024-05-17T15:08:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1250681856 bytes 2024-05-17T15:08:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1350122496 bytes 2024-05-17T15:08:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1368368128 bytes 2024-05-17T15:08:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1410953216 bytes 2024-05-17T15:08:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1464846336 bytes 2024-05-17T15:08:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1481039872 bytes 2024-05-17T15:08:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1512055808 bytes 2024-05-17T15:08:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1595854848 bytes 2024-05-17T15:08:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1651434496 bytes 2024-05-17T15:08:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1690567680 bytes 2024-05-17T15:08:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1717743616 bytes 2024-05-17T15:08:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1752031232 bytes 2024-05-17T15:08:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1856104448 bytes 2024-05-17T15:08:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1912793088 bytes 2024-05-17T15:08:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1976549376 bytes 2024-05-17T15:08:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1997707264 bytes 2024-05-17T15:08:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2011890688 bytes 2024-05-17T15:08:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2059053056 bytes 2024-05-17T15:08:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2089995264 bytes 2024-05-17T15:08:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2115346432 bytes 2024-05-17T15:08:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2165663744 bytes 2024-05-17T15:08:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2223276032 bytes 2024-05-17T15:08:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2284457984 bytes 2024-05-17T15:08:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2305381376 bytes 2024-05-17T15:08:29.865 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3558797215955265) > weight (0.25) 2024-05-17T15:08:29.866 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-05-17T15:08:29.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-05-17T15:08:29.867 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-05-17T15:08:29.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:08:29.866+0000 7f2814b7d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 48332) UID: 1000 2024-05-17T15:08:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2340880384 bytes 2024-05-17T15:08:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2355073024 bytes 2024-05-17T15:08:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2376205312 bytes 2024-05-17T15:08:30.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2400286720 bytes 2024-05-17T15:08:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2464105472 bytes 2024-05-17T15:08:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2476294144 bytes 2024-05-17T15:08:30.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2506977280 bytes 2024-05-17T15:08:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2555805696 bytes 2024-05-17T15:08:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2597903360 bytes 2024-05-17T15:08:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2620515328 bytes 2024-05-17T15:08:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:08:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:08:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:08:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:08:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:08:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_23 2024-05-17T15:08:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-05-17T15:08:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:08:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:08:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:08:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_23 128M 2024-05-17T15:08:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:08:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:08:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:08:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:08:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_23 2024-05-17T15:08:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:08:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:08:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:08:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-05-17T15:08:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:08:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:08:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-05-17T15:08:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:08:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-05-17T15:08:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_23': 2024-05-17T15:08:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:08:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:08:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:08:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1369e0e289b1 2024-05-17T15:08:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1369e0e289b1 2024-05-17T15:08:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:08:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:08:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:08:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:08:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:08:31 2024 2024-05-17T15:08:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:08:31 2024 2024-05-17T15:08:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:08:31 2024 2024-05-17T15:08:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:08:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:08:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 2968dbed-dc66-4e82-be5f-bc6e3322a6c1 2024-05-17T15:08:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:08:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:08:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_23 2024-05-17T15:08:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:08:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:08:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=14 2024-05-17T15:08:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:08:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:08:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 57342976 bytes 2024-05-17T15:08:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 76090368 bytes 2024-05-17T15:08:32.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 88331264 bytes 2024-05-17T15:08:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 122300416 bytes 2024-05-17T15:08:32.526 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 126090240 bytes 2024-05-17T15:08:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 142690304 bytes 2024-05-17T15:08:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 185546752 bytes 2024-05-17T15:08:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 243238912 bytes 2024-05-17T15:08:33.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 282260480 bytes 2024-05-17T15:08:33.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 282877952 bytes 2024-05-17T15:08:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 334640128 bytes 2024-05-17T15:08:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 352718848 bytes 2024-05-17T15:08:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 389511168 bytes 2024-05-17T15:08:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 432496640 bytes 2024-05-17T15:08:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 494951424 bytes 2024-05-17T15:08:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 566824960 bytes 2024-05-17T15:08:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 583709696 bytes 2024-05-17T15:08:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 616294400 bytes 2024-05-17T15:08:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 720175104 bytes 2024-05-17T15:08:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 756346880 bytes 2024-05-17T15:08:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 779390976 bytes 2024-05-17T15:08:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 836363264 bytes 2024-05-17T15:08:34.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 883843072 bytes 2024-05-17T15:08:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 920861696 bytes 2024-05-17T15:08:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 952561664 bytes 2024-05-17T15:08:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1050715136 bytes 2024-05-17T15:08:35.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1088259072 bytes 2024-05-17T15:08:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1094367232 bytes 2024-05-17T15:08:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1147684864 bytes 2024-05-17T15:08:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1206111232 bytes 2024-05-17T15:08:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1219207168 bytes 2024-05-17T15:08:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1268334592 bytes 2024-05-17T15:08:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1268967424 bytes 2024-05-17T15:08:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1270134784 bytes 2024-05-17T15:08:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1305421824 bytes 2024-05-17T15:08:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1360728064 bytes 2024-05-17T15:08:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1414040576 bytes 2024-05-17T15:08:36.868 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-05-17T15:08:36.868 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:08:36.868 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-05-17T15:08:36.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-05-17T15:08:36.869 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-05-17T15:08:36.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-05-17T15:08:36.872 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2024-05-17T15:08:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1479871488 bytes 2024-05-17T15:08:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1516508160 bytes 2024-05-17T15:08:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1552945152 bytes 2024-05-17T15:08:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1557470208 bytes 2024-05-17T15:08:37.489 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1588963328 bytes 2024-05-17T15:08:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1638064128 bytes 2024-05-17T15:08:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1664536576 bytes 2024-05-17T15:08:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1698788352 bytes 2024-05-17T15:08:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1743982592 bytes 2024-05-17T15:08:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1766264832 bytes 2024-05-17T15:08:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 1829515264 bytes 2024-05-17T15:08:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 1871412224 bytes 2024-05-17T15:08:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 1917321216 bytes 2024-05-17T15:08:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 1940364288 bytes 2024-05-17T15:08:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 1999716352 bytes 2024-05-17T15:08:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2002575360 bytes 2024-05-17T15:08:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2074388480 bytes 2024-05-17T15:08:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2086233088 bytes 2024-05-17T15:08:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2128980992 bytes 2024-05-17T15:08:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2176911360 bytes 2024-05-17T15:08:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2250295296 bytes 2024-05-17T15:08:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2312240128 bytes 2024-05-17T15:08:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2337611776 bytes 2024-05-17T15:08:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2404745216 bytes 2024-05-17T15:08:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2421056512 bytes 2024-05-17T15:08:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2481913856 bytes 2024-05-17T15:08:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2511549440 bytes 2024-05-17T15:08:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2568896512 bytes 2024-05-17T15:08:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2603212800 bytes 2024-05-17T15:08:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2641766400 bytes 2024-05-17T15:08:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2660557824 bytes 2024-05-17T15:08:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2722415616 bytes 2024-05-17T15:08:41.271 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 2747012096 bytes 2024-05-17T15:08:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 2831659008 bytes 2024-05-17T15:08:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 2898186240 bytes 2024-05-17T15:08:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 2929547264 bytes 2024-05-17T15:08:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 2957872128 bytes 2024-05-17T15:08:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 2970600448 bytes 2024-05-17T15:08:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3028055040 bytes 2024-05-17T15:08:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3090258944 bytes 2024-05-17T15:08:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3176643584 bytes 2024-05-17T15:08:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3245190144 bytes 2024-05-17T15:08:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3309956096 bytes 2024-05-17T15:08:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3395350528 bytes 2024-05-17T15:08:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3457274880 bytes 2024-05-17T15:08:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3492915200 bytes 2024-05-17T15:08:43.471 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3568716800 bytes 2024-05-17T15:08:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3733204992 bytes 2024-05-17T15:08:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3733801984 bytes 2024-05-17T15:08:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3776470016 bytes 2024-05-17T15:08:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3851249664 bytes 2024-05-17T15:08:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3878410240 bytes 2024-05-17T15:08:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3893656576 bytes 2024-05-17T15:08:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 3954178048 bytes 2024-05-17T15:08:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9200 IOs, 3969011712 bytes 2024-05-17T15:08:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9300 IOs, 3969645568 bytes 2024-05-17T15:08:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9400 IOs, 4025116672 bytes 2024-05-17T15:08:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9500 IOs, 4044243968 bytes 2024-05-17T15:08:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9600 IOs, 4058361856 bytes 2024-05-17T15:08:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9700 IOs, 4083093504 bytes 2024-05-17T15:08:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9800 IOs, 4090399744 bytes 2024-05-17T15:08:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9900 IOs, 4141345792 bytes 2024-05-17T15:08:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10000 IOs, 4178289664 bytes 2024-05-17T15:08:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10100 IOs, 4202296320 bytes 2024-05-17T15:08:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10200 IOs, 4202926080 bytes 2024-05-17T15:08:45.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10300 IOs, 4215915520 bytes 2024-05-17T15:08:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:08:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:08:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:08:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:08:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:08:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_24 2024-05-17T15:08:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-05-17T15:08:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:08:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:08:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:08:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_24 128M 2024-05-17T15:08:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:08:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:08:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:08:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:08:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_24 2024-05-17T15:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-05-17T15:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:08:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:08:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-05-17T15:08:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:08:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-05-17T15:08:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_24': 2024-05-17T15:08:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:08:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:08:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:08:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 13769c08382 2024-05-17T15:08:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.13769c08382 2024-05-17T15:08:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:08:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:08:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:08:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:08:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:08:45 2024 2024-05-17T15:08:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:08:45 2024 2024-05-17T15:08:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:08:45 2024 2024-05-17T15:08:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:08:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:08:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: d95d4f33-deca-446d-88ca-2fd4ef56f554 2024-05-17T15:08:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:08:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:08:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_24 2024-05-17T15:08:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:08:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:08:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=10 2024-05-17T15:08:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:08:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:08:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 26942464 bytes 2024-05-17T15:08:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 104687616 bytes 2024-05-17T15:08:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 146453504 bytes 2024-05-17T15:08:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 182324224 bytes 2024-05-17T15:08:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 274329600 bytes 2024-05-17T15:08:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 399024128 bytes 2024-05-17T15:08:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 430889984 bytes 2024-05-17T15:08:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 519065600 bytes 2024-05-17T15:08:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 527578112 bytes 2024-05-17T15:08:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 570390528 bytes 2024-05-17T15:08:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 647392256 bytes 2024-05-17T15:08:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 647980032 bytes 2024-05-17T15:08:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 705988608 bytes 2024-05-17T15:08:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 713779200 bytes 2024-05-17T15:08:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 747964416 bytes 2024-05-17T15:08:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 774583296 bytes 2024-05-17T15:08:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 871480320 bytes 2024-05-17T15:08:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 960499712 bytes 2024-05-17T15:08:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 961128448 bytes 2024-05-17T15:08:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 1027899392 bytes 2024-05-17T15:08:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 1100062720 bytes 2024-05-17T15:08:49.862 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1107545088 bytes 2024-05-17T15:08:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1122654208 bytes 2024-05-17T15:08:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1168676864 bytes 2024-05-17T15:08:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1209992192 bytes 2024-05-17T15:08:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1231367168 bytes 2024-05-17T15:08:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1279245312 bytes 2024-05-17T15:08:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1306768384 bytes 2024-05-17T15:08:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1387544576 bytes 2024-05-17T15:08:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1445713920 bytes 2024-05-17T15:08:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1465089024 bytes 2024-05-17T15:08:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1481292800 bytes 2024-05-17T15:08:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1525044224 bytes 2024-05-17T15:08:51.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1556849664 bytes 2024-05-17T15:08:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1600625664 bytes 2024-05-17T15:08:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1641337856 bytes 2024-05-17T15:08:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1680273408 bytes 2024-05-17T15:08:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1733258240 bytes 2024-05-17T15:08:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1817900032 bytes 2024-05-17T15:08:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1916698624 bytes 2024-05-17T15:08:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1949263872 bytes 2024-05-17T15:08:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1987876864 bytes 2024-05-17T15:08:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 2045073408 bytes 2024-05-17T15:08:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 2091597824 bytes 2024-05-17T15:08:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2151342080 bytes 2024-05-17T15:08:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2216892416 bytes 2024-05-17T15:08:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2260124672 bytes 2024-05-17T15:08:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2329658368 bytes 2024-05-17T15:08:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2354292736 bytes 2024-05-17T15:08:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2435258368 bytes 2024-05-17T15:08:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2450671616 bytes 2024-05-17T15:08:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2460223488 bytes 2024-05-17T15:08:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2505200640 bytes 2024-05-17T15:08:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2515612672 bytes 2024-05-17T15:08:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2564092928 bytes 2024-05-17T15:08:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2649492480 bytes 2024-05-17T15:08:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2678788096 bytes 2024-05-17T15:08:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2707278848 bytes 2024-05-17T15:08:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2756746240 bytes 2024-05-17T15:08:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2808243200 bytes 2024-05-17T15:08:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2864538624 bytes 2024-05-17T15:08:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2934521856 bytes 2024-05-17T15:08:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2960014336 bytes 2024-05-17T15:08:55.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 3022811136 bytes 2024-05-17T15:08:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 3077160960 bytes 2024-05-17T15:08:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 3111771136 bytes 2024-05-17T15:08:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 3174232064 bytes 2024-05-17T15:08:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 3210560512 bytes 2024-05-17T15:08:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 3265120256 bytes 2024-05-17T15:08:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:08:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:08:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:08:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:08:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:08:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_25 2024-05-17T15:08:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-05-17T15:08:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:08:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:08:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:08:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_25 128M 2024-05-17T15:08:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:08:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:08:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:08:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:08:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:08:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_25 2024-05-17T15:08:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:08:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:08:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:08:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-05-17T15:08:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:08:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:08:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-05-17T15:08:57.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:08:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-05-17T15:08:57.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_25': 2024-05-17T15:08:57.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:08:57.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:08:57.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:08:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 137fc8f4901 2024-05-17T15:08:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.137fc8f4901 2024-05-17T15:08:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:08:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:08:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:08:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:08:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:08:56 2024 2024-05-17T15:08:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:08:56 2024 2024-05-17T15:08:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:08:56 2024 2024-05-17T15:08:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:08:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:08:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 5f4f313b-d9df-420f-8c92-2d9e481184bc 2024-05-17T15:08:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:08:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:08:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_25 2024-05-17T15:08:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:08:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:08:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:08:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:08:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=13 2024-05-17T15:08:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:08:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:08:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 24240128 bytes 2024-05-17T15:08:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 103431168 bytes 2024-05-17T15:08:58.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 120394752 bytes 2024-05-17T15:08:58.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 154854400 bytes 2024-05-17T15:08:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 196157440 bytes 2024-05-17T15:08:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 292249600 bytes 2024-05-17T15:08:58.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 316075008 bytes 2024-05-17T15:08:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 372020224 bytes 2024-05-17T15:08:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 372613120 bytes 2024-05-17T15:08:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 425702400 bytes 2024-05-17T15:08:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 529406976 bytes 2024-05-17T15:08:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 577702912 bytes 2024-05-17T15:08:59.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 618179584 bytes 2024-05-17T15:08:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 662090752 bytes 2024-05-17T15:08:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 680479744 bytes 2024-05-17T15:08:59.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 757809152 bytes 2024-05-17T15:09:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 768587776 bytes 2024-05-17T15:09:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 773669888 bytes 2024-05-17T15:09:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 812210176 bytes 2024-05-17T15:09:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 873137152 bytes 2024-05-17T15:09:00.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 918369280 bytes 2024-05-17T15:09:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 923498496 bytes 2024-05-17T15:09:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1000864768 bytes 2024-05-17T15:09:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1025547264 bytes 2024-05-17T15:09:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1068762112 bytes 2024-05-17T15:09:01.329 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1155140608 bytes 2024-05-17T15:09:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1197739008 bytes 2024-05-17T15:09:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1268074496 bytes 2024-05-17T15:09:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1312507904 bytes 2024-05-17T15:09:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1319839744 bytes 2024-05-17T15:09:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1419491328 bytes 2024-05-17T15:09:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1425401856 bytes 2024-05-17T15:09:02.330 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1479992320 bytes 2024-05-17T15:09:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1531293696 bytes 2024-05-17T15:09:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1549492224 bytes 2024-05-17T15:09:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1631501312 bytes 2024-05-17T15:09:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1680002048 bytes 2024-05-17T15:09:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1729889280 bytes 2024-05-17T15:09:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1798977536 bytes 2024-05-17T15:09:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1815308288 bytes 2024-05-17T15:09:03.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1917050880 bytes 2024-05-17T15:09:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 2031846400 bytes 2024-05-17T15:09:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 2033953792 bytes 2024-05-17T15:09:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 2080444416 bytes 2024-05-17T15:09:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2088812544 bytes 2024-05-17T15:09:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2182025216 bytes 2024-05-17T15:09:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2240882688 bytes 2024-05-17T15:09:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2271003648 bytes 2024-05-17T15:09:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2283931648 bytes 2024-05-17T15:09:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2379776000 bytes 2024-05-17T15:09:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2486722560 bytes 2024-05-17T15:09:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2498520064 bytes 2024-05-17T15:09:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2549833728 bytes 2024-05-17T15:09:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2590824448 bytes 2024-05-17T15:09:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2641409024 bytes 2024-05-17T15:09:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2673720320 bytes 2024-05-17T15:09:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2675068928 bytes 2024-05-17T15:09:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2753006592 bytes 2024-05-17T15:09:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2756543488 bytes 2024-05-17T15:09:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2809426944 bytes 2024-05-17T15:09:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2832750592 bytes 2024-05-17T15:09:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2859476992 bytes 2024-05-17T15:09:06.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2917010432 bytes 2024-05-17T15:09:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2940985344 bytes 2024-05-17T15:09:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2966506496 bytes 2024-05-17T15:09:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2998859776 bytes 2024-05-17T15:09:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 3090432000 bytes 2024-05-17T15:09:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 3092563968 bytes 2024-05-17T15:09:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 3132267520 bytes 2024-05-17T15:09:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 3182034944 bytes 2024-05-17T15:09:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3197157376 bytes 2024-05-17T15:09:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3220595712 bytes 2024-05-17T15:09:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3254146048 bytes 2024-05-17T15:09:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3288515584 bytes 2024-05-17T15:09:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3318125568 bytes 2024-05-17T15:09:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3334128640 bytes 2024-05-17T15:09:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3368167424 bytes 2024-05-17T15:09:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3400763392 bytes 2024-05-17T15:09:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3449366528 bytes 2024-05-17T15:09:08.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3544795136 bytes 2024-05-17T15:09:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3585822720 bytes 2024-05-17T15:09:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3620474880 bytes 2024-05-17T15:09:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3642964992 bytes 2024-05-17T15:09:09.345 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3678296064 bytes 2024-05-17T15:09:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3748573184 bytes 2024-05-17T15:09:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3807284224 bytes 2024-05-17T15:09:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3821938688 bytes 2024-05-17T15:09:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3930881024 bytes 2024-05-17T15:09:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3931488256 bytes 2024-05-17T15:09:10.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3938148352 bytes 2024-05-17T15:09:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 4023596032 bytes 2024-05-17T15:09:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9200 IOs, 4073390080 bytes 2024-05-17T15:09:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:09:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:09:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:09:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:09:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:09:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_26 2024-05-17T15:09:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-05-17T15:09:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:09:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:09:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:09:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_26 128M 2024-05-17T15:09:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:09:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:09:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:09:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:09:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_26 2024-05-17T15:09:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:09:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:09:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:09:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-05-17T15:09:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:09:10.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:09:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-05-17T15:09:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:09:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-05-17T15:09:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_26': 2024-05-17T15:09:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:09:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:09:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:09:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1387f5772b9b 2024-05-17T15:09:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1387f5772b9b 2024-05-17T15:09:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:09:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:09:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:09:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:09:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:09:10 2024 2024-05-17T15:09:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:09:10 2024 2024-05-17T15:09:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:09:10 2024 2024-05-17T15:09:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:09:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:09:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 9d50b403-8aa5-4da6-8e72-4a3f61c9358b 2024-05-17T15:09:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:09:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:09:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_26 2024-05-17T15:09:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:09:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:09:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=13 2024-05-17T15:09:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:09:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:09:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 56138752 bytes 2024-05-17T15:09:11.880 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8295252084762993) > weight (0.25) 2024-05-17T15:09:11.881 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.800052969265388) > weight (0.25) 2024-05-17T15:09:11.881 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6327540677860128) > weight (0.25) 2024-05-17T15:09:11.881 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.40605578342183923) > weight (0.25) 2024-05-17T15:09:11.881 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-05-17T15:09:11.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 101870592 bytes 2024-05-17T15:09:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 102693888 bytes 2024-05-17T15:09:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 166874112 bytes 2024-05-17T15:09:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 225234944 bytes 2024-05-17T15:09:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 243555328 bytes 2024-05-17T15:09:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 290097152 bytes 2024-05-17T15:09:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 327254016 bytes 2024-05-17T15:09:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 396588032 bytes 2024-05-17T15:09:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 439187456 bytes 2024-05-17T15:09:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 504709120 bytes 2024-05-17T15:09:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 517026816 bytes 2024-05-17T15:09:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 585446400 bytes 2024-05-17T15:09:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 646910976 bytes 2024-05-17T15:09:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 710033408 bytes 2024-05-17T15:09:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 725923840 bytes 2024-05-17T15:09:14.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 794255360 bytes 2024-05-17T15:09:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 794869760 bytes 2024-05-17T15:09:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 795480064 bytes 2024-05-17T15:09:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 849239040 bytes 2024-05-17T15:09:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 891150336 bytes 2024-05-17T15:09:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 948840448 bytes 2024-05-17T15:09:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1010719744 bytes 2024-05-17T15:09:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1047521280 bytes 2024-05-17T15:09:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1048145920 bytes 2024-05-17T15:09:15.156 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1063326720 bytes 2024-05-17T15:09:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1108843520 bytes 2024-05-17T15:09:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1151094784 bytes 2024-05-17T15:09:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1164144640 bytes 2024-05-17T15:09:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1192103936 bytes 2024-05-17T15:09:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1213164544 bytes 2024-05-17T15:09:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1219008512 bytes 2024-05-17T15:09:15.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1238742016 bytes 2024-05-17T15:09:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1289544704 bytes 2024-05-17T15:09:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1312167936 bytes 2024-05-17T15:09:16.323 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1360713728 bytes 2024-05-17T15:09:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1362828288 bytes 2024-05-17T15:09:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1411207168 bytes 2024-05-17T15:09:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1442528256 bytes 2024-05-17T15:09:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1484305408 bytes 2024-05-17T15:09:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1505476608 bytes 2024-05-17T15:09:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1531424768 bytes 2024-05-17T15:09:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1592410112 bytes 2024-05-17T15:09:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1625912320 bytes 2024-05-17T15:09:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1659552768 bytes 2024-05-17T15:09:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1716379648 bytes 2024-05-17T15:09:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1765949440 bytes 2024-05-17T15:09:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 1816311808 bytes 2024-05-17T15:09:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 1850124288 bytes 2024-05-17T15:09:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 1850721280 bytes 2024-05-17T15:09:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 1862778880 bytes 2024-05-17T15:09:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 1948238848 bytes 2024-05-17T15:09:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2031331328 bytes 2024-05-17T15:09:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2043520000 bytes 2024-05-17T15:09:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2044113920 bytes 2024-05-17T15:09:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2076153856 bytes 2024-05-17T15:09:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2168857600 bytes 2024-05-17T15:09:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2249689088 bytes 2024-05-17T15:09:19.378 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2252206080 bytes 2024-05-17T15:09:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2253270016 bytes 2024-05-17T15:09:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2316134400 bytes 2024-05-17T15:09:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2353872896 bytes 2024-05-17T15:09:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2432686080 bytes 2024-05-17T15:09:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2484320256 bytes 2024-05-17T15:09:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2518841344 bytes 2024-05-17T15:09:20.300 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:09:20.298+0000 7f7a4dbdb640 -1 reset not still connected to 0x55d8de3f0b60 2024-05-17T15:09:20.300 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:09:20.298+0000 7f7a4dbdb640 -1 reset not still connected to 0x55d8e2715ad0 2024-05-17T15:09:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2589770752 bytes 2024-05-17T15:09:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2607790080 bytes 2024-05-17T15:09:20.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2680900608 bytes 2024-05-17T15:09:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2736430080 bytes 2024-05-17T15:09:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 2743973888 bytes 2024-05-17T15:09:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 2769460224 bytes 2024-05-17T15:09:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 2807274496 bytes 2024-05-17T15:09:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 2879506432 bytes 2024-05-17T15:09:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 2935280640 bytes 2024-05-17T15:09:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 2966962176 bytes 2024-05-17T15:09:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 2976064512 bytes 2024-05-17T15:09:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3006682112 bytes 2024-05-17T15:09:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3031949312 bytes 2024-05-17T15:09:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3102651392 bytes 2024-05-17T15:09:22.185 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3138519040 bytes 2024-05-17T15:09:22.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3190716416 bytes 2024-05-17T15:09:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3254626304 bytes 2024-05-17T15:09:22.596 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3277515776 bytes 2024-05-17T15:09:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3300910080 bytes 2024-05-17T15:09:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3308149760 bytes 2024-05-17T15:09:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3381522432 bytes 2024-05-17T15:09:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3465106432 bytes 2024-05-17T15:09:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3506244608 bytes 2024-05-17T15:09:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3573905408 bytes 2024-05-17T15:09:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3601448960 bytes 2024-05-17T15:09:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 3677690880 bytes 2024-05-17T15:09:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9200 IOs, 3699892224 bytes 2024-05-17T15:09:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9300 IOs, 3803872256 bytes 2024-05-17T15:09:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9400 IOs, 3830145024 bytes 2024-05-17T15:09:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9500 IOs, 3878323200 bytes 2024-05-17T15:09:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9600 IOs, 3895813120 bytes 2024-05-17T15:09:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9700 IOs, 3956853760 bytes 2024-05-17T15:09:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:09:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:09:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:09:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:09:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:09:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_27 2024-05-17T15:09:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-05-17T15:09:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_27 2024-05-17T15:09:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:09:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:09:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_27 128M 2024-05-17T15:09:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_27 2024-05-17T15:09:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:09:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:09:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:09:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_27 2024-05-17T15:09:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:09:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:09:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:09:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-05-17T15:09:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_27 2024-05-17T15:09:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:09:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-05-17T15:09:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:09:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-05-17T15:09:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_27': 2024-05-17T15:09:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:09:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:09:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:09:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 13923a6d931e 2024-05-17T15:09:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.13923a6d931e 2024-05-17T15:09:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:09:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:09:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:09:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:09:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:09:24 2024 2024-05-17T15:09:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:09:24 2024 2024-05-17T15:09:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:09:24 2024 2024-05-17T15:09:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:09:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:09:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: d0b80867-2a3b-4947-ae2d-30cd6b4ea8cc 2024-05-17T15:09:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:09:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:09:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_27 2024-05-17T15:09:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_27 2024-05-17T15:09:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:09:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=14 2024-05-17T15:09:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:09:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:09:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 53080064 bytes 2024-05-17T15:09:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 95230976 bytes 2024-05-17T15:09:26.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 148348928 bytes 2024-05-17T15:09:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 157093888 bytes 2024-05-17T15:09:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 213693440 bytes 2024-05-17T15:09:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 336569344 bytes 2024-05-17T15:09:26.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 384063488 bytes 2024-05-17T15:09:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 448982016 bytes 2024-05-17T15:09:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 494951424 bytes 2024-05-17T15:09:27.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 599405568 bytes 2024-05-17T15:09:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 600011776 bytes 2024-05-17T15:09:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 624374784 bytes 2024-05-17T15:09:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 654552064 bytes 2024-05-17T15:09:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 672705536 bytes 2024-05-17T15:09:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 724829184 bytes 2024-05-17T15:09:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 725902336 bytes 2024-05-17T15:09:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 740240384 bytes 2024-05-17T15:09:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 796475392 bytes 2024-05-17T15:09:28.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 797091840 bytes 2024-05-17T15:09:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 813892608 bytes 2024-05-17T15:09:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 845889536 bytes 2024-05-17T15:09:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 890650624 bytes 2024-05-17T15:09:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 992926720 bytes 2024-05-17T15:09:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1062620160 bytes 2024-05-17T15:09:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1108530176 bytes 2024-05-17T15:09:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1164303360 bytes 2024-05-17T15:09:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1187663872 bytes 2024-05-17T15:09:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1249335296 bytes 2024-05-17T15:09:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1283801088 bytes 2024-05-17T15:09:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1319073792 bytes 2024-05-17T15:09:30.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1371686912 bytes 2024-05-17T15:09:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1387229184 bytes 2024-05-17T15:09:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1407920128 bytes 2024-05-17T15:09:30.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1466682368 bytes 2024-05-17T15:09:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1535332352 bytes 2024-05-17T15:09:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1552275456 bytes 2024-05-17T15:09:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1583048704 bytes 2024-05-17T15:09:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1645603840 bytes 2024-05-17T15:09:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1652730880 bytes 2024-05-17T15:09:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1678070784 bytes 2024-05-17T15:09:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1722136576 bytes 2024-05-17T15:09:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1745519616 bytes 2024-05-17T15:09:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1774349312 bytes 2024-05-17T15:09:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1834608640 bytes 2024-05-17T15:09:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1875896320 bytes 2024-05-17T15:09:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1952324608 bytes 2024-05-17T15:09:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2016227328 bytes 2024-05-17T15:09:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2052529152 bytes 2024-05-17T15:09:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2082578432 bytes 2024-05-17T15:09:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2105580544 bytes 2024-05-17T15:09:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2113309696 bytes 2024-05-17T15:09:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2142366720 bytes 2024-05-17T15:09:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2169973760 bytes 2024-05-17T15:09:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2220985344 bytes 2024-05-17T15:09:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2304279552 bytes 2024-05-17T15:09:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2334538752 bytes 2024-05-17T15:09:33.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2346446848 bytes 2024-05-17T15:09:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2395409408 bytes 2024-05-17T15:09:33.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2463216640 bytes 2024-05-17T15:09:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2463841280 bytes 2024-05-17T15:09:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2476001280 bytes 2024-05-17T15:09:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2476598272 bytes 2024-05-17T15:09:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2522728448 bytes 2024-05-17T15:09:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2589172736 bytes 2024-05-17T15:09:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2636899328 bytes 2024-05-17T15:09:34.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2694781952 bytes 2024-05-17T15:09:34.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2695408640 bytes 2024-05-17T15:09:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2715109376 bytes 2024-05-17T15:09:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2771073024 bytes 2024-05-17T15:09:35.340 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 2795838464 bytes 2024-05-17T15:09:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 2870984704 bytes 2024-05-17T15:09:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 2921957376 bytes 2024-05-17T15:09:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 2981232640 bytes 2024-05-17T15:09:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 2981854208 bytes 2024-05-17T15:09:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3006770176 bytes 2024-05-17T15:09:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3034472448 bytes 2024-05-17T15:09:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3115671552 bytes 2024-05-17T15:09:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3139172352 bytes 2024-05-17T15:09:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3230094336 bytes 2024-05-17T15:09:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3330055168 bytes 2024-05-17T15:09:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3340227584 bytes 2024-05-17T15:09:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3383221248 bytes 2024-05-17T15:09:37.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3510665216 bytes 2024-05-17T15:09:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3552242688 bytes 2024-05-17T15:09:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3607973888 bytes 2024-05-17T15:09:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3672188928 bytes 2024-05-17T15:09:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3723386880 bytes 2024-05-17T15:09:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3755823104 bytes 2024-05-17T15:09:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3792584704 bytes 2024-05-17T15:09:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3888024576 bytes 2024-05-17T15:09:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 3932569600 bytes 2024-05-17T15:09:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9200 IOs, 3953693696 bytes 2024-05-17T15:09:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9300 IOs, 4010636288 bytes 2024-05-17T15:09:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9400 IOs, 4082044928 bytes 2024-05-17T15:09:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9500 IOs, 4110613504 bytes 2024-05-17T15:09:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9600 IOs, 4135279616 bytes 2024-05-17T15:09:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9700 IOs, 4215469056 bytes 2024-05-17T15:09:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9800 IOs, 4266457088 bytes 2024-05-17T15:09:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:09:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:09:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:09:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:09:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:09:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_28 2024-05-17T15:09:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-05-17T15:09:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_28 2024-05-17T15:09:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:09:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:09:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_28 128M 2024-05-17T15:09:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_28 2024-05-17T15:09:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:09:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:09:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:09:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_28 2024-05-17T15:09:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:09:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:09:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:09:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-05-17T15:09:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_28 2024-05-17T15:09:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:09:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-05-17T15:09:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:09:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-05-17T15:09:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_28': 2024-05-17T15:09:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:09:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:09:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:09:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 139be6dedfcb 2024-05-17T15:09:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.139be6dedfcb 2024-05-17T15:09:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:09:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:09:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:09:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:09:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:09:39 2024 2024-05-17T15:09:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:09:39 2024 2024-05-17T15:09:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:09:39 2024 2024-05-17T15:09:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:09:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:09:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 90f08d14-8ef0-4bff-9f0d-d804ed59aad1 2024-05-17T15:09:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:09:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:09:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_28 2024-05-17T15:09:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_28 2024-05-17T15:09:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:09:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=5 2024-05-17T15:09:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:09:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:09:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 2790400 bytes 2024-05-17T15:09:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 39901184 bytes 2024-05-17T15:09:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 103279616 bytes 2024-05-17T15:09:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 147920896 bytes 2024-05-17T15:09:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 173992960 bytes 2024-05-17T15:09:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 175976448 bytes 2024-05-17T15:09:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 262105088 bytes 2024-05-17T15:09:41.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 306526208 bytes 2024-05-17T15:09:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 308468736 bytes 2024-05-17T15:09:41.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 329531392 bytes 2024-05-17T15:09:41.329 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 370048000 bytes 2024-05-17T15:09:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 402161664 bytes 2024-05-17T15:09:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 457199616 bytes 2024-05-17T15:09:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 457817088 bytes 2024-05-17T15:09:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 505005056 bytes 2024-05-17T15:09:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 509705216 bytes 2024-05-17T15:09:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 549484544 bytes 2024-05-17T15:09:42.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 588087296 bytes 2024-05-17T15:09:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 641406976 bytes 2024-05-17T15:09:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 659530752 bytes 2024-05-17T15:09:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 670170112 bytes 2024-05-17T15:09:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 710950912 bytes 2024-05-17T15:09:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 744868864 bytes 2024-05-17T15:09:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 867383296 bytes 2024-05-17T15:09:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 951413760 bytes 2024-05-17T15:09:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 964709376 bytes 2024-05-17T15:09:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1014689792 bytes 2024-05-17T15:09:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1026840576 bytes 2024-05-17T15:09:43.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1081878528 bytes 2024-05-17T15:09:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1123995648 bytes 2024-05-17T15:09:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1182475264 bytes 2024-05-17T15:09:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1221874688 bytes 2024-05-17T15:09:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1260301312 bytes 2024-05-17T15:09:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1290022912 bytes 2024-05-17T15:09:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1312801792 bytes 2024-05-17T15:09:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1398934528 bytes 2024-05-17T15:09:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:09:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:09:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:09:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:09:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:09:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_29 2024-05-17T15:09:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-05-17T15:09:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_29 2024-05-17T15:09:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:09:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:09:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_29 128M 2024-05-17T15:09:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_29 2024-05-17T15:09:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:09:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:09:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:09:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_29 2024-05-17T15:09:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:09:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:09:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:09:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-05-17T15:09:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_29 2024-05-17T15:09:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:09:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-05-17T15:09:45.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:09:45.860 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-05-17T15:09:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_29': 2024-05-17T15:09:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:09:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:09:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:09:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 13a41fae1f57 2024-05-17T15:09:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.13a41fae1f57 2024-05-17T15:09:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:09:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:09:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:09:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:09:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:09:44 2024 2024-05-17T15:09:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:09:44 2024 2024-05-17T15:09:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:09:44 2024 2024-05-17T15:09:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:09:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:09:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 59928e42-fdf9-4cdf-a1c9-1cabd73a05c5 2024-05-17T15:09:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:09:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:09:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_29 2024-05-17T15:09:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_29 2024-05-17T15:09:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:09:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=12 2024-05-17T15:09:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:09:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:09:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 21356544 bytes 2024-05-17T15:09:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 21948416 bytes 2024-05-17T15:09:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 58642432 bytes 2024-05-17T15:09:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 108634112 bytes 2024-05-17T15:09:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 148098048 bytes 2024-05-17T15:09:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 182965248 bytes 2024-05-17T15:09:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 219828224 bytes 2024-05-17T15:09:46.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 298387456 bytes 2024-05-17T15:09:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 303564800 bytes 2024-05-17T15:09:47.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 376452096 bytes 2024-05-17T15:09:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 395523072 bytes 2024-05-17T15:09:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 433645568 bytes 2024-05-17T15:09:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 485347328 bytes 2024-05-17T15:09:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 502052864 bytes 2024-05-17T15:09:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 565491712 bytes 2024-05-17T15:09:48.023 INFO:tasks.ceph.cluster2.osd.1.smithi195.stderr:2024-05-17T15:09:48.021+0000 7f839c24e640 -1 reset not still connected to 0x5616d49ed2b0 2024-05-17T15:09:48.024 INFO:tasks.ceph.cluster2.osd.1.smithi195.stderr:2024-05-17T15:09:48.021+0000 7f839c24e640 -1 reset not still connected to 0x5616e264c0d0 2024-05-17T15:09:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 579028992 bytes 2024-05-17T15:09:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 649075712 bytes 2024-05-17T15:09:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 689862656 bytes 2024-05-17T15:09:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 747041792 bytes 2024-05-17T15:09:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 800822272 bytes 2024-05-17T15:09:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 816587776 bytes 2024-05-17T15:09:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 889588736 bytes 2024-05-17T15:09:48.971 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:09:48.969+0000 7f7a4dbdb640 -1 reset not still connected to 0x55d8f5e6ca90 2024-05-17T15:09:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 890208256 bytes 2024-05-17T15:09:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 948012032 bytes 2024-05-17T15:09:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1010500608 bytes 2024-05-17T15:09:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1036077056 bytes 2024-05-17T15:09:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1056855040 bytes 2024-05-17T15:09:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1133237248 bytes 2024-05-17T15:09:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1189101568 bytes 2024-05-17T15:09:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1206372352 bytes 2024-05-17T15:09:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1269493760 bytes 2024-05-17T15:09:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1283636224 bytes 2024-05-17T15:09:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1326647296 bytes 2024-05-17T15:09:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1359208448 bytes 2024-05-17T15:09:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1392752640 bytes 2024-05-17T15:09:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1418158080 bytes 2024-05-17T15:09:50.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1457468416 bytes 2024-05-17T15:09:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1547246592 bytes 2024-05-17T15:09:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1595339776 bytes 2024-05-17T15:09:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1620012032 bytes 2024-05-17T15:09:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1688942592 bytes 2024-05-17T15:09:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1745971200 bytes 2024-05-17T15:09:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1780756480 bytes 2024-05-17T15:09:51.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1810697216 bytes 2024-05-17T15:09:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1814587392 bytes 2024-05-17T15:09:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1860255744 bytes 2024-05-17T15:09:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1912581120 bytes 2024-05-17T15:09:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 1950509056 bytes 2024-05-17T15:09:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 1951122432 bytes 2024-05-17T15:09:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 1979400192 bytes 2024-05-17T15:09:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2003030016 bytes 2024-05-17T15:09:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2041830400 bytes 2024-05-17T15:09:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2109371392 bytes 2024-05-17T15:09:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2157446144 bytes 2024-05-17T15:09:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2209879040 bytes 2024-05-17T15:09:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2234722304 bytes 2024-05-17T15:09:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2282427392 bytes 2024-05-17T15:09:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2305367040 bytes 2024-05-17T15:09:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2347149312 bytes 2024-05-17T15:09:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2389879808 bytes 2024-05-17T15:09:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2418491392 bytes 2024-05-17T15:09:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2448534528 bytes 2024-05-17T15:09:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2460363776 bytes 2024-05-17T15:09:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2510846976 bytes 2024-05-17T15:09:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2617716736 bytes 2024-05-17T15:09:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2654109696 bytes 2024-05-17T15:09:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2678313984 bytes 2024-05-17T15:09:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2723466240 bytes 2024-05-17T15:09:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2765547520 bytes 2024-05-17T15:09:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 2812493824 bytes 2024-05-17T15:09:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 2823285760 bytes 2024-05-17T15:09:55.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 2823930880 bytes 2024-05-17T15:09:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 2830483456 bytes 2024-05-17T15:09:55.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 2841398272 bytes 2024-05-17T15:09:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 2888512512 bytes 2024-05-17T15:09:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 2919966720 bytes 2024-05-17T15:09:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 2989672448 bytes 2024-05-17T15:09:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3036907520 bytes 2024-05-17T15:09:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3094780928 bytes 2024-05-17T15:09:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3127175168 bytes 2024-05-17T15:09:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3127821312 bytes 2024-05-17T15:09:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3151527936 bytes 2024-05-17T15:09:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3262179328 bytes 2024-05-17T15:09:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3278625792 bytes 2024-05-17T15:09:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3296125952 bytes 2024-05-17T15:09:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3320312832 bytes 2024-05-17T15:09:57.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3351354368 bytes 2024-05-17T15:09:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3366566912 bytes 2024-05-17T15:09:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3397866496 bytes 2024-05-17T15:09:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3411771392 bytes 2024-05-17T15:09:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:09:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:09:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:09:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:09:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:09:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_30 2024-05-17T15:09:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-05-17T15:09:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_30 2024-05-17T15:09:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:09:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:09:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_30 128M 2024-05-17T15:09:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_30 2024-05-17T15:09:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:09:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:09:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:09:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:09:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_30 2024-05-17T15:09:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:09:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:09:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:09:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-05-17T15:09:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_30 2024-05-17T15:09:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:09:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-05-17T15:09:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:09:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-05-17T15:09:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_30': 2024-05-17T15:09:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:09:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:09:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:09:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 13ad1c749d1f 2024-05-17T15:09:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.13ad1c749d1f 2024-05-17T15:09:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:09:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:09:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:09:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:09:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:09:57 2024 2024-05-17T15:09:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:09:57 2024 2024-05-17T15:09:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:09:57 2024 2024-05-17T15:09:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:09:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:09:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 2031e3c2-db04-4d88-a9f5-2a8ddf1e9fce 2024-05-17T15:09:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:09:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:09:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_30 2024-05-17T15:09:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:09:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:09:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_30 2024-05-17T15:09:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:09:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=12 2024-05-17T15:09:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:09:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:09:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 52813824 bytes 2024-05-17T15:09:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 138021888 bytes 2024-05-17T15:09:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 177380352 bytes 2024-05-17T15:09:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 239702016 bytes 2024-05-17T15:09:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 286108672 bytes 2024-05-17T15:09:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 326528000 bytes 2024-05-17T15:09:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 351363072 bytes 2024-05-17T15:09:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 444705792 bytes 2024-05-17T15:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 445322240 bytes 2024-05-17T15:10:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 495360000 bytes 2024-05-17T15:10:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 528108544 bytes 2024-05-17T15:10:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 528714752 bytes 2024-05-17T15:10:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 583414784 bytes 2024-05-17T15:10:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 705129472 bytes 2024-05-17T15:10:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 705742848 bytes 2024-05-17T15:10:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 762216448 bytes 2024-05-17T15:10:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 773770240 bytes 2024-05-17T15:10:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 811052032 bytes 2024-05-17T15:10:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 855420928 bytes 2024-05-17T15:10:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 913709056 bytes 2024-05-17T15:10:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 975226880 bytes 2024-05-17T15:10:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1027144704 bytes 2024-05-17T15:10:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1047659520 bytes 2024-05-17T15:10:02.105 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1135532032 bytes 2024-05-17T15:10:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1162809344 bytes 2024-05-17T15:10:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1248239616 bytes 2024-05-17T15:10:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1320825856 bytes 2024-05-17T15:10:02.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1362958336 bytes 2024-05-17T15:10:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1373443072 bytes 2024-05-17T15:10:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1435933696 bytes 2024-05-17T15:10:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1472838656 bytes 2024-05-17T15:10:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1524494336 bytes 2024-05-17T15:10:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1565986816 bytes 2024-05-17T15:10:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1670107136 bytes 2024-05-17T15:10:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1695325184 bytes 2024-05-17T15:10:03.873 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-05-17T15:10:03.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-05-17T15:10:03.874 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-05-17T15:10:03.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T15:10:03.873+0000 7f3668aa9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 41112) UID: 1000 2024-05-17T15:10:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1752365056 bytes 2024-05-17T15:10:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1766847488 bytes 2024-05-17T15:10:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1870546944 bytes 2024-05-17T15:10:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1885267968 bytes 2024-05-17T15:10:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1904432128 bytes 2024-05-17T15:10:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1946658816 bytes 2024-05-17T15:10:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 2001609728 bytes 2024-05-17T15:10:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 2084976640 bytes 2024-05-17T15:10:05.113 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 2103976960 bytes 2024-05-17T15:10:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2164446208 bytes 2024-05-17T15:10:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2237134848 bytes 2024-05-17T15:10:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2256490496 bytes 2024-05-17T15:10:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2294443008 bytes 2024-05-17T15:10:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2325365760 bytes 2024-05-17T15:10:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2375339008 bytes 2024-05-17T15:10:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2396018688 bytes 2024-05-17T15:10:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2396619776 bytes 2024-05-17T15:10:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2480263168 bytes 2024-05-17T15:10:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2535025664 bytes 2024-05-17T15:10:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2603967488 bytes 2024-05-17T15:10:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2625728512 bytes 2024-05-17T15:10:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2653451264 bytes 2024-05-17T15:10:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2688407552 bytes 2024-05-17T15:10:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2744898560 bytes 2024-05-17T15:10:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2791479296 bytes 2024-05-17T15:10:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2827151360 bytes 2024-05-17T15:10:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2872968192 bytes 2024-05-17T15:10:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2901864448 bytes 2024-05-17T15:10:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2954398720 bytes 2024-05-17T15:10:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 3014656000 bytes 2024-05-17T15:10:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 3035915264 bytes 2024-05-17T15:10:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 3134655488 bytes 2024-05-17T15:10:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 3211292672 bytes 2024-05-17T15:10:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 3293814784 bytes 2024-05-17T15:10:08.875 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-05-17T15:10:08.875 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:10:08.875 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-05-17T15:10:08.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-05-17T15:10:08.876 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-05-17T15:10:08.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-05-17T15:10:08.879 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-05-17T15:10:08.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 3345638400 bytes 2024-05-17T15:10:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3359911936 bytes 2024-05-17T15:10:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3380430848 bytes 2024-05-17T15:10:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3439029248 bytes 2024-05-17T15:10:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3495181312 bytes 2024-05-17T15:10:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3545745408 bytes 2024-05-17T15:10:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3546358784 bytes 2024-05-17T15:10:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3614020608 bytes 2024-05-17T15:10:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3650746368 bytes 2024-05-17T15:10:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3709510656 bytes 2024-05-17T15:10:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3723164672 bytes 2024-05-17T15:10:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3768242176 bytes 2024-05-17T15:10:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:10:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:10:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:10:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:10:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:10:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_31 2024-05-17T15:10:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-05-17T15:10:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_31 2024-05-17T15:10:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:10:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:10:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_31 128M 2024-05-17T15:10:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_31 2024-05-17T15:10:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:10:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:10:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:10:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_31 2024-05-17T15:10:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:10:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:10:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:10:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-05-17T15:10:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_31 2024-05-17T15:10:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:10:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-05-17T15:10:11.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:10:11.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-05-17T15:10:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_31': 2024-05-17T15:10:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:10:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:10:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:10:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 13bc8454796 2024-05-17T15:10:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.13bc8454796 2024-05-17T15:10:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:10:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:10:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:10:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:10:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:10:10 2024 2024-05-17T15:10:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:10:10 2024 2024-05-17T15:10:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:10:10 2024 2024-05-17T15:10:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:10:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:10:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 219fe1bd-503a-4377-b187-f83062f5b1ba 2024-05-17T15:10:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:10:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:10:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_31 2024-05-17T15:10:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_31 2024-05-17T15:10:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:10:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=14 2024-05-17T15:10:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:10:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:10:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 94249984 bytes 2024-05-17T15:10:11.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 101659648 bytes 2024-05-17T15:10:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 108759040 bytes 2024-05-17T15:10:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 209610752 bytes 2024-05-17T15:10:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 308839424 bytes 2024-05-17T15:10:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 361201664 bytes 2024-05-17T15:10:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 401078272 bytes 2024-05-17T15:10:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 485977088 bytes 2024-05-17T15:10:13.123 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 555518976 bytes 2024-05-17T15:10:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 674446336 bytes 2024-05-17T15:10:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 702120960 bytes 2024-05-17T15:10:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 742614016 bytes 2024-05-17T15:10:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 802783232 bytes 2024-05-17T15:10:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 838332416 bytes 2024-05-17T15:10:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 858782720 bytes 2024-05-17T15:10:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 909591552 bytes 2024-05-17T15:10:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 966674432 bytes 2024-05-17T15:10:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 1054422016 bytes 2024-05-17T15:10:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 1088906240 bytes 2024-05-17T15:10:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 1132225536 bytes 2024-05-17T15:10:14.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 1202305024 bytes 2024-05-17T15:10:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1202922496 bytes 2024-05-17T15:10:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1244020736 bytes 2024-05-17T15:10:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1280698368 bytes 2024-05-17T15:10:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1321319424 bytes 2024-05-17T15:10:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1348632576 bytes 2024-05-17T15:10:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1367372800 bytes 2024-05-17T15:10:15.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1428114432 bytes 2024-05-17T15:10:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1428715520 bytes 2024-05-17T15:10:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1459661824 bytes 2024-05-17T15:10:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1502643200 bytes 2024-05-17T15:10:16.277 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1554784256 bytes 2024-05-17T15:10:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1625062400 bytes 2024-05-17T15:10:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1653428224 bytes 2024-05-17T15:10:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1705919488 bytes 2024-05-17T15:10:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1741853696 bytes 2024-05-17T15:10:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1763561472 bytes 2024-05-17T15:10:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1785489408 bytes 2024-05-17T15:10:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1819378688 bytes 2024-05-17T15:10:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1850403840 bytes 2024-05-17T15:10:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1881759744 bytes 2024-05-17T15:10:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1934770176 bytes 2024-05-17T15:10:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1952757760 bytes 2024-05-17T15:10:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1975871488 bytes 2024-05-17T15:10:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1978401792 bytes 2024-05-17T15:10:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2029086720 bytes 2024-05-17T15:10:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2057924608 bytes 2024-05-17T15:10:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2172363776 bytes 2024-05-17T15:10:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2227260416 bytes 2024-05-17T15:10:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2281782272 bytes 2024-05-17T15:10:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2321087488 bytes 2024-05-17T15:10:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2333312000 bytes 2024-05-17T15:10:19.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2373364736 bytes 2024-05-17T15:10:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2445760512 bytes 2024-05-17T15:10:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2457719808 bytes 2024-05-17T15:10:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2479342592 bytes 2024-05-17T15:10:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2610667520 bytes 2024-05-17T15:10:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2652122112 bytes 2024-05-17T15:10:19.882 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4125726654232982) > weight (0.25) 2024-05-17T15:10:19.882 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7715481341391602) > weight (0.25) 2024-05-17T15:10:19.882 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-05-17T15:10:19.883 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-05-17T15:10:19.883 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-05-17T15:10:19.883 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-05-17T15:10:19.883 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:10:19.884 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi195.stderr:2024-05-17T15:10:19.881+0000 7f548d218640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 42203) UID: 1000 2024-05-17T15:10:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2700309504 bytes 2024-05-17T15:10:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2796098560 bytes 2024-05-17T15:10:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2812161024 bytes 2024-05-17T15:10:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2872891392 bytes 2024-05-17T15:10:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2911406080 bytes 2024-05-17T15:10:20.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2984970240 bytes 2024-05-17T15:10:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 3034259456 bytes 2024-05-17T15:10:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 3071902720 bytes 2024-05-17T15:10:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 3118112768 bytes 2024-05-17T15:10:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 3187043328 bytes 2024-05-17T15:10:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 3258321920 bytes 2024-05-17T15:10:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 3298676736 bytes 2024-05-17T15:10:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3417296896 bytes 2024-05-17T15:10:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3417917440 bytes 2024-05-17T15:10:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3490779136 bytes 2024-05-17T15:10:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3549694976 bytes 2024-05-17T15:10:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3614044160 bytes 2024-05-17T15:10:22.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3664194560 bytes 2024-05-17T15:10:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3678016512 bytes 2024-05-17T15:10:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3735399424 bytes 2024-05-17T15:10:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3818010624 bytes 2024-05-17T15:10:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3847579648 bytes 2024-05-17T15:10:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3915548672 bytes 2024-05-17T15:10:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3993234432 bytes 2024-05-17T15:10:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 4042619904 bytes 2024-05-17T15:10:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 4071561216 bytes 2024-05-17T15:10:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 4117709824 bytes 2024-05-17T15:10:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 4127438848 bytes 2024-05-17T15:10:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 4155322368 bytes 2024-05-17T15:10:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 4251734016 bytes 2024-05-17T15:10:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 4294336512 bytes 2024-05-17T15:10:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 4378045440 bytes 2024-05-17T15:10:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 4417634304 bytes 2024-05-17T15:10:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9200 IOs, 4478852096 bytes 2024-05-17T15:10:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9300 IOs, 4490004480 bytes 2024-05-17T15:10:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9400 IOs, 4503355392 bytes 2024-05-17T15:10:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9500 IOs, 4535273472 bytes 2024-05-17T15:10:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9600 IOs, 4559264768 bytes 2024-05-17T15:10:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9700 IOs, 4570726400 bytes 2024-05-17T15:10:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:10:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:10:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:10:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:10:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:10:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_32 2024-05-17T15:10:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-05-17T15:10:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_32 2024-05-17T15:10:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:10:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:10:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_32 128M 2024-05-17T15:10:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_32 2024-05-17T15:10:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:10:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:10:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:10:25.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:25.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_32 2024-05-17T15:10:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:10:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:10:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:10:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-05-17T15:10:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_32 2024-05-17T15:10:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:10:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-05-17T15:10:25.985 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-05-17T15:10:25.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-05-17T15:10:25.985 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-05-17T15:10:25.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-05-17T15:10:25.989 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-05-17T15:10:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:10:26.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-05-17T15:10:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_32': 2024-05-17T15:10:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:10:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:10:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:10:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 13c5b76bd901 2024-05-17T15:10:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.13c5b76bd901 2024-05-17T15:10:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:10:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:10:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:10:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:10:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:10:25 2024 2024-05-17T15:10:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:10:25 2024 2024-05-17T15:10:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:10:25 2024 2024-05-17T15:10:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:10:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:10:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 0835fff5-1be2-449a-8ff6-85a5ed6b71f8 2024-05-17T15:10:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:10:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:10:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_32 2024-05-17T15:10:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_32 2024-05-17T15:10:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:10:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=8 2024-05-17T15:10:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:10:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:10:26.860 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 33999872 bytes 2024-05-17T15:10:26.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 75929600 bytes 2024-05-17T15:10:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 80730112 bytes 2024-05-17T15:10:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 105693184 bytes 2024-05-17T15:10:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 126205952 bytes 2024-05-17T15:10:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 201724928 bytes 2024-05-17T15:10:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 257949696 bytes 2024-05-17T15:10:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 325878784 bytes 2024-05-17T15:10:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 364167168 bytes 2024-05-17T15:10:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 436962304 bytes 2024-05-17T15:10:28.351 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 537785344 bytes 2024-05-17T15:10:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 592595968 bytes 2024-05-17T15:10:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 614641664 bytes 2024-05-17T15:10:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 639190016 bytes 2024-05-17T15:10:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 696809472 bytes 2024-05-17T15:10:29.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 756265984 bytes 2024-05-17T15:10:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 785323008 bytes 2024-05-17T15:10:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 845930496 bytes 2024-05-17T15:10:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 911516672 bytes 2024-05-17T15:10:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 965949440 bytes 2024-05-17T15:10:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 1027164160 bytes 2024-05-17T15:10:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1083269120 bytes 2024-05-17T15:10:30.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1093993472 bytes 2024-05-17T15:10:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1156057088 bytes 2024-05-17T15:10:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1178980352 bytes 2024-05-17T15:10:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1242974208 bytes 2024-05-17T15:10:30.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1250569216 bytes 2024-05-17T15:10:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1256591360 bytes 2024-05-17T15:10:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1257208832 bytes 2024-05-17T15:10:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1292433408 bytes 2024-05-17T15:10:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1382029312 bytes 2024-05-17T15:10:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1399313408 bytes 2024-05-17T15:10:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1436394496 bytes 2024-05-17T15:10:31.492 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1487142912 bytes 2024-05-17T15:10:31.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1511495680 bytes 2024-05-17T15:10:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1530466304 bytes 2024-05-17T15:10:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1587952640 bytes 2024-05-17T15:10:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1627880448 bytes 2024-05-17T15:10:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1713923072 bytes 2024-05-17T15:10:32.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1734279168 bytes 2024-05-17T15:10:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1789948928 bytes 2024-05-17T15:10:32.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1815264256 bytes 2024-05-17T15:10:32.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1846292480 bytes 2024-05-17T15:10:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1931997184 bytes 2024-05-17T15:10:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1966828544 bytes 2024-05-17T15:10:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2027618304 bytes 2024-05-17T15:10:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2063588352 bytes 2024-05-17T15:10:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2122559488 bytes 2024-05-17T15:10:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2207966208 bytes 2024-05-17T15:10:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2263829504 bytes 2024-05-17T15:10:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2334336000 bytes 2024-05-17T15:10:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2360225792 bytes 2024-05-17T15:10:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2366462976 bytes 2024-05-17T15:10:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2393062400 bytes 2024-05-17T15:10:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2425820160 bytes 2024-05-17T15:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_33 2024-05-17T15:10:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-05-17T15:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_33 2024-05-17T15:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:10:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:10:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_33 128M 2024-05-17T15:10:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_33 2024-05-17T15:10:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:10:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:10:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:10:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_33 2024-05-17T15:10:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:10:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:10:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:10:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-05-17T15:10:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_33 2024-05-17T15:10:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:10:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-05-17T15:10:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:10:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-05-17T15:10:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_33': 2024-05-17T15:10:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:10:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:10:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:10:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 13db2eaeeb24 2024-05-17T15:10:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.13db2eaeeb24 2024-05-17T15:10:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:10:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:10:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:10:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:10:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:10:34 2024 2024-05-17T15:10:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:10:34 2024 2024-05-17T15:10:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:10:34 2024 2024-05-17T15:10:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:10:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:10:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 0218530c-9285-4e1b-ac61-baf8583aceb9 2024-05-17T15:10:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:10:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:10:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_33 2024-05-17T15:10:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_33 2024-05-17T15:10:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:10:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=6 2024-05-17T15:10:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:10:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:10:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 47838208 bytes 2024-05-17T15:10:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 91918336 bytes 2024-05-17T15:10:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 113167360 bytes 2024-05-17T15:10:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 117893120 bytes 2024-05-17T15:10:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 118507520 bytes 2024-05-17T15:10:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 121537536 bytes 2024-05-17T15:10:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 146401280 bytes 2024-05-17T15:10:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 182204416 bytes 2024-05-17T15:10:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 225188864 bytes 2024-05-17T15:10:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 228669440 bytes 2024-05-17T15:10:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 256624640 bytes 2024-05-17T15:10:37.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 272392192 bytes 2024-05-17T15:10:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 325583872 bytes 2024-05-17T15:10:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 343226368 bytes 2024-05-17T15:10:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 437648384 bytes 2024-05-17T15:10:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 472401920 bytes 2024-05-17T15:10:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 500887552 bytes 2024-05-17T15:10:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 531028992 bytes 2024-05-17T15:10:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 605988864 bytes 2024-05-17T15:10:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 663911424 bytes 2024-05-17T15:10:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 686781440 bytes 2024-05-17T15:10:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 720049152 bytes 2024-05-17T15:10:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 762512384 bytes 2024-05-17T15:10:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 795339776 bytes 2024-05-17T15:10:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 857796608 bytes 2024-05-17T15:10:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 872377344 bytes 2024-05-17T15:10:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 944075776 bytes 2024-05-17T15:10:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 982434816 bytes 2024-05-17T15:10:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 995703808 bytes 2024-05-17T15:10:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1076078592 bytes 2024-05-17T15:10:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1142579200 bytes 2024-05-17T15:10:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1156288512 bytes 2024-05-17T15:10:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1183404032 bytes 2024-05-17T15:10:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1271742464 bytes 2024-05-17T15:10:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1289531392 bytes 2024-05-17T15:10:40.523 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1314485248 bytes 2024-05-17T15:10:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1367068672 bytes 2024-05-17T15:10:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1391768576 bytes 2024-05-17T15:10:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1465725952 bytes 2024-05-17T15:10:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1503565824 bytes 2024-05-17T15:10:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1599922176 bytes 2024-05-17T15:10:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1649400832 bytes 2024-05-17T15:10:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1664936960 bytes 2024-05-17T15:10:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:10:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:10:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:10:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:10:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:10:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_34 2024-05-17T15:10:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-05-17T15:10:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_34 2024-05-17T15:10:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:10:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:10:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_34 128M 2024-05-17T15:10:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_34 2024-05-17T15:10:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:10:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:10:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:10:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_34 2024-05-17T15:10:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:10:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:10:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:10:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-05-17T15:10:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_34 2024-05-17T15:10:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:10:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-05-17T15:10:42.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:10:42.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-05-17T15:10:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_34': 2024-05-17T15:10:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:10:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:10:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:10:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 13e77ef0af88 2024-05-17T15:10:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.13e77ef0af88 2024-05-17T15:10:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:10:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:10:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:10:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:10:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:10:41 2024 2024-05-17T15:10:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:10:41 2024 2024-05-17T15:10:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:10:41 2024 2024-05-17T15:10:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:10:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:10:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 32af895d-8b5f-4d18-99db-b2ad45a7ac69 2024-05-17T15:10:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:10:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:10:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_34 2024-05-17T15:10:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_34 2024-05-17T15:10:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:10:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=10 2024-05-17T15:10:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:10:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:10:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 608256 bytes 2024-05-17T15:10:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 30628864 bytes 2024-05-17T15:10:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 40866816 bytes 2024-05-17T15:10:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 87073792 bytes 2024-05-17T15:10:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 176500736 bytes 2024-05-17T15:10:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 234109952 bytes 2024-05-17T15:10:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 302407680 bytes 2024-05-17T15:10:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 347585536 bytes 2024-05-17T15:10:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 384463872 bytes 2024-05-17T15:10:44.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 389033984 bytes 2024-05-17T15:10:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 423280640 bytes 2024-05-17T15:10:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 479254528 bytes 2024-05-17T15:10:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 563200000 bytes 2024-05-17T15:10:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 617614336 bytes 2024-05-17T15:10:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 675372032 bytes 2024-05-17T15:10:45.200 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 707769344 bytes 2024-05-17T15:10:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 767437824 bytes 2024-05-17T15:10:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 792881152 bytes 2024-05-17T15:10:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 802518016 bytes 2024-05-17T15:10:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 836552704 bytes 2024-05-17T15:10:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 884028416 bytes 2024-05-17T15:10:45.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 920140800 bytes 2024-05-17T15:10:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1027891200 bytes 2024-05-17T15:10:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1028515840 bytes 2024-05-17T15:10:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1084519424 bytes 2024-05-17T15:10:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1127015424 bytes 2024-05-17T15:10:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1207341056 bytes 2024-05-17T15:10:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1238836224 bytes 2024-05-17T15:10:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1245916160 bytes 2024-05-17T15:10:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1287507968 bytes 2024-05-17T15:10:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1366989824 bytes 2024-05-17T15:10:47.431 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1392905216 bytes 2024-05-17T15:10:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1413822464 bytes 2024-05-17T15:10:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1447822336 bytes 2024-05-17T15:10:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1486484480 bytes 2024-05-17T15:10:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1574050816 bytes 2024-05-17T15:10:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1611128832 bytes 2024-05-17T15:10:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1622953984 bytes 2024-05-17T15:10:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1659201536 bytes 2024-05-17T15:10:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1685097472 bytes 2024-05-17T15:10:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1722810368 bytes 2024-05-17T15:10:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1800786944 bytes 2024-05-17T15:10:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1845391360 bytes 2024-05-17T15:10:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1866006528 bytes 2024-05-17T15:10:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1913576448 bytes 2024-05-17T15:10:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1927375872 bytes 2024-05-17T15:10:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1962699776 bytes 2024-05-17T15:10:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2061457408 bytes 2024-05-17T15:10:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2090899456 bytes 2024-05-17T15:10:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2113572864 bytes 2024-05-17T15:10:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2167759872 bytes 2024-05-17T15:10:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2201859072 bytes 2024-05-17T15:10:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2210540544 bytes 2024-05-17T15:10:50.471 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2270182400 bytes 2024-05-17T15:10:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2376388608 bytes 2024-05-17T15:10:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2394172416 bytes 2024-05-17T15:10:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2449106944 bytes 2024-05-17T15:10:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2449704960 bytes 2024-05-17T15:10:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2486120448 bytes 2024-05-17T15:10:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2486740992 bytes 2024-05-17T15:10:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2500412416 bytes 2024-05-17T15:10:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2624604160 bytes 2024-05-17T15:10:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2673998848 bytes 2024-05-17T15:10:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2727681024 bytes 2024-05-17T15:10:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2789805056 bytes 2024-05-17T15:10:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2862923776 bytes 2024-05-17T15:10:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2873892864 bytes 2024-05-17T15:10:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2903640064 bytes 2024-05-17T15:10:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2959312896 bytes 2024-05-17T15:10:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:10:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:10:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:10:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:10:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:10:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_35 2024-05-17T15:10:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-05-17T15:10:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_35 2024-05-17T15:10:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:10:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:10:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_35 128M 2024-05-17T15:10:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_35 2024-05-17T15:10:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:10:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:10:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:10:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:10:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_35 2024-05-17T15:10:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:10:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:10:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:10:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-05-17T15:10:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_35 2024-05-17T15:10:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:10:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-05-17T15:10:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:10:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-05-17T15:10:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_35': 2024-05-17T15:10:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:10:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:10:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:10:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 13f3e8decfb5 2024-05-17T15:10:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.13f3e8decfb5 2024-05-17T15:10:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:10:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:10:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:10:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:10:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:10:52 2024 2024-05-17T15:10:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:10:52 2024 2024-05-17T15:10:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:10:52 2024 2024-05-17T15:10:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:10:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:10:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 8c639172-6810-4a20-a5ee-19b5a50ec6fd 2024-05-17T15:10:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:10:53.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:10:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_35 2024-05-17T15:10:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:10:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:10:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_35 2024-05-17T15:10:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:10:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=13 2024-05-17T15:10:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:10:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:10:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 49781760 bytes 2024-05-17T15:10:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 147133440 bytes 2024-05-17T15:10:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 147750912 bytes 2024-05-17T15:10:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 177951744 bytes 2024-05-17T15:10:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 194212864 bytes 2024-05-17T15:10:54.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 226484224 bytes 2024-05-17T15:10:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 254871552 bytes 2024-05-17T15:10:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 282931200 bytes 2024-05-17T15:10:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 329909248 bytes 2024-05-17T15:10:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 427816960 bytes 2024-05-17T15:10:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 504420352 bytes 2024-05-17T15:10:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 549620736 bytes 2024-05-17T15:10:55.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 556173312 bytes 2024-05-17T15:10:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 580568064 bytes 2024-05-17T15:10:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 584928256 bytes 2024-05-17T15:10:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 585539584 bytes 2024-05-17T15:10:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 638512128 bytes 2024-05-17T15:10:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 675105792 bytes 2024-05-17T15:10:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 723575808 bytes 2024-05-17T15:10:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 754424832 bytes 2024-05-17T15:10:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 778386432 bytes 2024-05-17T15:10:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 832164864 bytes 2024-05-17T15:10:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 857866240 bytes 2024-05-17T15:10:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 930253824 bytes 2024-05-17T15:10:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 989113344 bytes 2024-05-17T15:10:57.388 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1047137280 bytes 2024-05-17T15:10:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1077091328 bytes 2024-05-17T15:10:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1159383040 bytes 2024-05-17T15:10:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1193951232 bytes 2024-05-17T15:10:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1263683584 bytes 2024-05-17T15:10:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1327840256 bytes 2024-05-17T15:10:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1372624896 bytes 2024-05-17T15:10:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1418105856 bytes 2024-05-17T15:10:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1439273984 bytes 2024-05-17T15:10:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1523271680 bytes 2024-05-17T15:10:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1587473408 bytes 2024-05-17T15:10:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1649575936 bytes 2024-05-17T15:10:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1728314368 bytes 2024-05-17T15:10:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1787302912 bytes 2024-05-17T15:10:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1817039872 bytes 2024-05-17T15:10:59.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1849752576 bytes 2024-05-17T15:10:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1876358144 bytes 2024-05-17T15:10:59.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1887534080 bytes 2024-05-17T15:11:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1920296960 bytes 2024-05-17T15:11:00.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1947297792 bytes 2024-05-17T15:11:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1995936768 bytes 2024-05-17T15:11:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2059018240 bytes 2024-05-17T15:11:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2127468544 bytes 2024-05-17T15:11:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2186825728 bytes 2024-05-17T15:11:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2206994432 bytes 2024-05-17T15:11:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2207601664 bytes 2024-05-17T15:11:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2270264320 bytes 2024-05-17T15:11:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2298884096 bytes 2024-05-17T15:11:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2311604224 bytes 2024-05-17T15:11:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2352025600 bytes 2024-05-17T15:11:01.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2413213696 bytes 2024-05-17T15:11:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2431293440 bytes 2024-05-17T15:11:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2462085120 bytes 2024-05-17T15:11:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2476043264 bytes 2024-05-17T15:11:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2489605120 bytes 2024-05-17T15:11:02.260 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2551983104 bytes 2024-05-17T15:11:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2577350656 bytes 2024-05-17T15:11:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2660470784 bytes 2024-05-17T15:11:02.696 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2704970752 bytes 2024-05-17T15:11:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2770342912 bytes 2024-05-17T15:11:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2776864768 bytes 2024-05-17T15:11:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2811986944 bytes 2024-05-17T15:11:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2852645888 bytes 2024-05-17T15:11:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2899211264 bytes 2024-05-17T15:11:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 3007332352 bytes 2024-05-17T15:11:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3019593728 bytes 2024-05-17T15:11:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3035123712 bytes 2024-05-17T15:11:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3064685568 bytes 2024-05-17T15:11:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3127619584 bytes 2024-05-17T15:11:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3201278976 bytes 2024-05-17T15:11:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3235196928 bytes 2024-05-17T15:11:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3266429952 bytes 2024-05-17T15:11:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3286378496 bytes 2024-05-17T15:11:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3298172928 bytes 2024-05-17T15:11:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3360699392 bytes 2024-05-17T15:11:05.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3427939328 bytes 2024-05-17T15:11:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3477994496 bytes 2024-05-17T15:11:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3526209536 bytes 2024-05-17T15:11:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3597678592 bytes 2024-05-17T15:11:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3649913856 bytes 2024-05-17T15:11:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3732876288 bytes 2024-05-17T15:11:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3817984000 bytes 2024-05-17T15:11:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3870871552 bytes 2024-05-17T15:11:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3917037568 bytes 2024-05-17T15:11:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3974665216 bytes 2024-05-17T15:11:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:11:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:11:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:11:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:11:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:11:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_36 2024-05-17T15:11:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-05-17T15:11:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_36 2024-05-17T15:11:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:11:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:11:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_36 128M 2024-05-17T15:11:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_36 2024-05-17T15:11:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:11:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:11:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:11:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_36 2024-05-17T15:11:06.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:11:06.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:06.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:11:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:11:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-05-17T15:11:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_36 2024-05-17T15:11:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:11:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-05-17T15:11:07.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:11:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-05-17T15:11:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_36': 2024-05-17T15:11:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:11:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:11:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:11:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 13ff84ba2e16 2024-05-17T15:11:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.13ff84ba2e16 2024-05-17T15:11:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:11:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:11:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:11:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:11:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:11:06 2024 2024-05-17T15:11:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:11:06 2024 2024-05-17T15:11:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:11:06 2024 2024-05-17T15:11:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:11:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:11:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: b8165b54-ecda-4ca3-8d0a-f0c1a85112c7 2024-05-17T15:11:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:11:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:11:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_36 2024-05-17T15:11:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_36 2024-05-17T15:11:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:11:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=14 2024-05-17T15:11:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:11:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:11:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 618496 bytes 2024-05-17T15:11:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 80863232 bytes 2024-05-17T15:11:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 118996992 bytes 2024-05-17T15:11:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 153156608 bytes 2024-05-17T15:11:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 195138560 bytes 2024-05-17T15:11:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 227117056 bytes 2024-05-17T15:11:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 242827264 bytes 2024-05-17T15:11:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 275343360 bytes 2024-05-17T15:11:09.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 301044736 bytes 2024-05-17T15:11:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 343604224 bytes 2024-05-17T15:11:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 376145920 bytes 2024-05-17T15:11:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 376749056 bytes 2024-05-17T15:11:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 439071744 bytes 2024-05-17T15:11:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 467013632 bytes 2024-05-17T15:11:09.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 523529216 bytes 2024-05-17T15:11:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 569854976 bytes 2024-05-17T15:11:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 570508288 bytes 2024-05-17T15:11:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 578115584 bytes 2024-05-17T15:11:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 601962496 bytes 2024-05-17T15:11:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 648578048 bytes 2024-05-17T15:11:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 713415680 bytes 2024-05-17T15:11:10.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 726083584 bytes 2024-05-17T15:11:10.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 737204224 bytes 2024-05-17T15:11:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 757922816 bytes 2024-05-17T15:11:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 822008832 bytes 2024-05-17T15:11:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 891829248 bytes 2024-05-17T15:11:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 945013760 bytes 2024-05-17T15:11:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1040578560 bytes 2024-05-17T15:11:11.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1106839552 bytes 2024-05-17T15:11:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1170820096 bytes 2024-05-17T15:11:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1234976768 bytes 2024-05-17T15:11:12.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1292141568 bytes 2024-05-17T15:11:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1316418560 bytes 2024-05-17T15:11:12.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1366565888 bytes 2024-05-17T15:11:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1458812928 bytes 2024-05-17T15:11:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1459870720 bytes 2024-05-17T15:11:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1487656960 bytes 2024-05-17T15:11:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1511514112 bytes 2024-05-17T15:11:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1544677376 bytes 2024-05-17T15:11:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1577099264 bytes 2024-05-17T15:11:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1624152064 bytes 2024-05-17T15:11:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1636442112 bytes 2024-05-17T15:11:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1689479168 bytes 2024-05-17T15:11:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1775373312 bytes 2024-05-17T15:11:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1846713344 bytes 2024-05-17T15:11:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1884731392 bytes 2024-05-17T15:11:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1906731008 bytes 2024-05-17T15:11:14.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 1936249856 bytes 2024-05-17T15:11:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2017657856 bytes 2024-05-17T15:11:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2031368192 bytes 2024-05-17T15:11:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2031976448 bytes 2024-05-17T15:11:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2097221632 bytes 2024-05-17T15:11:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2126299136 bytes 2024-05-17T15:11:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2155942912 bytes 2024-05-17T15:11:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2206773248 bytes 2024-05-17T15:11:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2245342208 bytes 2024-05-17T15:11:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2300712960 bytes 2024-05-17T15:11:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2305089536 bytes 2024-05-17T15:11:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2343958528 bytes 2024-05-17T15:11:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2388475904 bytes 2024-05-17T15:11:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2416424960 bytes 2024-05-17T15:11:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2486004736 bytes 2024-05-17T15:11:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2542573568 bytes 2024-05-17T15:11:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2566180864 bytes 2024-05-17T15:11:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2573271040 bytes 2024-05-17T15:11:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2657261568 bytes 2024-05-17T15:11:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2715549696 bytes 2024-05-17T15:11:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2775443456 bytes 2024-05-17T15:11:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2854064128 bytes 2024-05-17T15:11:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 2875325440 bytes 2024-05-17T15:11:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3009871872 bytes 2024-05-17T15:11:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3067245568 bytes 2024-05-17T15:11:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3119248384 bytes 2024-05-17T15:11:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3150158848 bytes 2024-05-17T15:11:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3155439616 bytes 2024-05-17T15:11:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3173703680 bytes 2024-05-17T15:11:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3235189760 bytes 2024-05-17T15:11:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3259399168 bytes 2024-05-17T15:11:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3276487680 bytes 2024-05-17T15:11:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3287144448 bytes 2024-05-17T15:11:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3338347520 bytes 2024-05-17T15:11:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3385447424 bytes 2024-05-17T15:11:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3430960128 bytes 2024-05-17T15:11:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3462695936 bytes 2024-05-17T15:11:19.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3507567616 bytes 2024-05-17T15:11:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3555225600 bytes 2024-05-17T15:11:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3611882496 bytes 2024-05-17T15:11:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3656719360 bytes 2024-05-17T15:11:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3723697152 bytes 2024-05-17T15:11:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3785937920 bytes 2024-05-17T15:11:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 3812855808 bytes 2024-05-17T15:11:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9200 IOs, 3822305280 bytes 2024-05-17T15:11:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9300 IOs, 3866718208 bytes 2024-05-17T15:11:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9400 IOs, 3873891328 bytes 2024-05-17T15:11:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9500 IOs, 3927452672 bytes 2024-05-17T15:11:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9600 IOs, 3976052736 bytes 2024-05-17T15:11:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9700 IOs, 3989270528 bytes 2024-05-17T15:11:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9800 IOs, 4009328640 bytes 2024-05-17T15:11:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9900 IOs, 4053417984 bytes 2024-05-17T15:11:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10000 IOs, 4124085248 bytes 2024-05-17T15:11:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10100 IOs, 4138783744 bytes 2024-05-17T15:11:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:11:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:11:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:11:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:11:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:11:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_37 2024-05-17T15:11:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-05-17T15:11:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_37 2024-05-17T15:11:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:11:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:11:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_37 128M 2024-05-17T15:11:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_37 2024-05-17T15:11:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:11:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:11:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:11:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_37 2024-05-17T15:11:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:11:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:11:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:11:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-05-17T15:11:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_37 2024-05-17T15:11:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:11:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-05-17T15:11:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:11:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-05-17T15:11:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_37': 2024-05-17T15:11:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:11:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:11:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:11:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 140b6b8ee275 2024-05-17T15:11:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.140b6b8ee275 2024-05-17T15:11:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:11:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:11:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:11:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:11:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:11:22 2024 2024-05-17T15:11:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:11:22 2024 2024-05-17T15:11:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:11:22 2024 2024-05-17T15:11:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:11:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:11:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 1cbb263d-bbcf-4037-8f80-52c0b1dfd284 2024-05-17T15:11:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:11:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:11:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_37 2024-05-17T15:11:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_37 2024-05-17T15:11:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:11:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=8 2024-05-17T15:11:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:11:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:11:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 38112256 bytes 2024-05-17T15:11:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 70187008 bytes 2024-05-17T15:11:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 141893632 bytes 2024-05-17T15:11:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 194248704 bytes 2024-05-17T15:11:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 244116480 bytes 2024-05-17T15:11:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 251617280 bytes 2024-05-17T15:11:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 340569088 bytes 2024-05-17T15:11:24.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 459994112 bytes 2024-05-17T15:11:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 477649920 bytes 2024-05-17T15:11:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 561494016 bytes 2024-05-17T15:11:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 611302400 bytes 2024-05-17T15:11:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 651773952 bytes 2024-05-17T15:11:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 663431168 bytes 2024-05-17T15:11:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 703905792 bytes 2024-05-17T15:11:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 741092352 bytes 2024-05-17T15:11:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 776972288 bytes 2024-05-17T15:11:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 836060160 bytes 2024-05-17T15:11:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 865909760 bytes 2024-05-17T15:11:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 951100416 bytes 2024-05-17T15:11:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 1011915776 bytes 2024-05-17T15:11:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 1090663424 bytes 2024-05-17T15:11:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1120716800 bytes 2024-05-17T15:11:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1161493504 bytes 2024-05-17T15:11:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1196307456 bytes 2024-05-17T15:11:26.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1265868800 bytes 2024-05-17T15:11:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1301472256 bytes 2024-05-17T15:11:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1336068096 bytes 2024-05-17T15:11:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1363870720 bytes 2024-05-17T15:11:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1394724864 bytes 2024-05-17T15:11:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1457067008 bytes 2024-05-17T15:11:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1457679360 bytes 2024-05-17T15:11:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1519260672 bytes 2024-05-17T15:11:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1522758656 bytes 2024-05-17T15:11:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1604376576 bytes 2024-05-17T15:11:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1605006336 bytes 2024-05-17T15:11:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1624152064 bytes 2024-05-17T15:11:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1649138688 bytes 2024-05-17T15:11:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1681677312 bytes 2024-05-17T15:11:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1749307392 bytes 2024-05-17T15:11:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1799617536 bytes 2024-05-17T15:11:29.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1800232960 bytes 2024-05-17T15:11:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1829195776 bytes 2024-05-17T15:11:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1887309824 bytes 2024-05-17T15:11:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1971980288 bytes 2024-05-17T15:11:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2054021120 bytes 2024-05-17T15:11:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2080770048 bytes 2024-05-17T15:11:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2123220992 bytes 2024-05-17T15:11:30.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2150473728 bytes 2024-05-17T15:11:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2195207168 bytes 2024-05-17T15:11:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2196752384 bytes 2024-05-17T15:11:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2237972480 bytes 2024-05-17T15:11:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2255961088 bytes 2024-05-17T15:11:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2280221696 bytes 2024-05-17T15:11:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2314831872 bytes 2024-05-17T15:11:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:11:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:11:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:11:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:11:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:11:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_38 2024-05-17T15:11:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-05-17T15:11:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_38 2024-05-17T15:11:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:11:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:11:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_38 128M 2024-05-17T15:11:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_38 2024-05-17T15:11:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:11:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:11:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:11:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_38 2024-05-17T15:11:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:11:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:11:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:11:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-05-17T15:11:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_38 2024-05-17T15:11:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:11:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-05-17T15:11:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:11:31.832 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-05-17T15:11:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_38': 2024-05-17T15:11:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:11:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:11:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:11:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1417ed459749 2024-05-17T15:11:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1417ed459749 2024-05-17T15:11:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:11:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:11:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:11:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:11:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:11:30 2024 2024-05-17T15:11:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:11:30 2024 2024-05-17T15:11:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:11:30 2024 2024-05-17T15:11:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:11:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:11:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: f990d936-a771-483b-8728-fd1503201136 2024-05-17T15:11:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:11:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:11:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_38 2024-05-17T15:11:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_38 2024-05-17T15:11:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:11:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=11 2024-05-17T15:11:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:11:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:11:31.989 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6838086952022173) > weight (0.25) 2024-05-17T15:11:31.990 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5957779522137273) > weight (0.25) 2024-05-17T15:11:31.990 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.856555884332522) > weight (0.25) 2024-05-17T15:11:31.990 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5312395763788529) > weight (0.25) 2024-05-17T15:11:31.990 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2024-05-17T15:11:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 55982080 bytes 2024-05-17T15:11:32.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 72580096 bytes 2024-05-17T15:11:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 119709696 bytes 2024-05-17T15:11:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 147069952 bytes 2024-05-17T15:11:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 192446464 bytes 2024-05-17T15:11:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 232487936 bytes 2024-05-17T15:11:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 287142912 bytes 2024-05-17T15:11:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 297247744 bytes 2024-05-17T15:11:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 391934976 bytes 2024-05-17T15:11:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 471794688 bytes 2024-05-17T15:11:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 513440768 bytes 2024-05-17T15:11:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 551477248 bytes 2024-05-17T15:11:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 593064960 bytes 2024-05-17T15:11:33.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 593682432 bytes 2024-05-17T15:11:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 663666688 bytes 2024-05-17T15:11:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 711469056 bytes 2024-05-17T15:11:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 781659136 bytes 2024-05-17T15:11:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 811070464 bytes 2024-05-17T15:11:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 883381248 bytes 2024-05-17T15:11:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 925783040 bytes 2024-05-17T15:11:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 1003474944 bytes 2024-05-17T15:11:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1026452480 bytes 2024-05-17T15:11:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1056769024 bytes 2024-05-17T15:11:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1074236416 bytes 2024-05-17T15:11:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1075465216 bytes 2024-05-17T15:11:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1095045120 bytes 2024-05-17T15:11:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1144546304 bytes 2024-05-17T15:11:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1209617408 bytes 2024-05-17T15:11:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1296414720 bytes 2024-05-17T15:11:36.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1320711168 bytes 2024-05-17T15:11:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1350621184 bytes 2024-05-17T15:11:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1376980992 bytes 2024-05-17T15:11:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1468399616 bytes 2024-05-17T15:11:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1510281216 bytes 2024-05-17T15:11:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1575638016 bytes 2024-05-17T15:11:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1611714560 bytes 2024-05-17T15:11:37.237 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1668892672 bytes 2024-05-17T15:11:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1705995264 bytes 2024-05-17T15:11:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1793352704 bytes 2024-05-17T15:11:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1846769664 bytes 2024-05-17T15:11:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1869662208 bytes 2024-05-17T15:11:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1889606656 bytes 2024-05-17T15:11:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1950688256 bytes 2024-05-17T15:11:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1968501760 bytes 2024-05-17T15:11:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2020413440 bytes 2024-05-17T15:11:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2064019456 bytes 2024-05-17T15:11:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2137441280 bytes 2024-05-17T15:11:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2214992896 bytes 2024-05-17T15:11:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2296769536 bytes 2024-05-17T15:11:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2343779328 bytes 2024-05-17T15:11:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2430722048 bytes 2024-05-17T15:11:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2518984704 bytes 2024-05-17T15:11:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2660910080 bytes 2024-05-17T15:11:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2684336128 bytes 2024-05-17T15:11:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2770927616 bytes 2024-05-17T15:11:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2814360576 bytes 2024-05-17T15:11:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2852793344 bytes 2024-05-17T15:11:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2922822656 bytes 2024-05-17T15:11:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2923450368 bytes 2024-05-17T15:11:40.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2950752256 bytes 2024-05-17T15:11:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2996868096 bytes 2024-05-17T15:11:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2997473280 bytes 2024-05-17T15:11:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 3085706240 bytes 2024-05-17T15:11:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 3136301056 bytes 2024-05-17T15:11:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 3196301312 bytes 2024-05-17T15:11:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 3206746112 bytes 2024-05-17T15:11:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 3309412352 bytes 2024-05-17T15:11:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 3370905600 bytes 2024-05-17T15:11:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 3411318784 bytes 2024-05-17T15:11:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 3513292800 bytes 2024-05-17T15:11:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3549433856 bytes 2024-05-17T15:11:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:11:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:11:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:11:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:11:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:11:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_39 2024-05-17T15:11:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-05-17T15:11:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_39 2024-05-17T15:11:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:11:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:11:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_39 128M 2024-05-17T15:11:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_39 2024-05-17T15:11:42.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:11:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:11:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:11:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_39 2024-05-17T15:11:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:11:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:11:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:11:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-05-17T15:11:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_39 2024-05-17T15:11:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:11:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-05-17T15:11:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:11:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-05-17T15:11:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_39': 2024-05-17T15:11:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:11:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:11:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:11:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1423e5a8d83c 2024-05-17T15:11:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1423e5a8d83c 2024-05-17T15:11:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:11:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:11:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:11:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:11:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:11:42 2024 2024-05-17T15:11:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:11:42 2024 2024-05-17T15:11:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:11:42 2024 2024-05-17T15:11:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:11:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:11:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: aae21142-bf9f-4d97-9ad7-8e4679ee037a 2024-05-17T15:11:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:11:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:11:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_39 2024-05-17T15:11:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_39 2024-05-17T15:11:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:11:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=5 2024-05-17T15:11:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:11:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:11:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 35094528 bytes 2024-05-17T15:11:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 77890560 bytes 2024-05-17T15:11:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 134872064 bytes 2024-05-17T15:11:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 176865280 bytes 2024-05-17T15:11:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 225808384 bytes 2024-05-17T15:11:44.839 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 280650752 bytes 2024-05-17T15:11:44.880 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4377512392150009) > weight (0.25) 2024-05-17T15:11:44.881 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9383104327809823) > weight (0.25) 2024-05-17T15:11:44.881 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3479109652783764) > weight (0.25) 2024-05-17T15:11:44.881 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.4789952712958434) > weight (0.25) 2024-05-17T15:11:44.881 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2024-05-17T15:11:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 286042112 bytes 2024-05-17T15:11:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 384019456 bytes 2024-05-17T15:11:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 427108352 bytes 2024-05-17T15:11:45.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 474198016 bytes 2024-05-17T15:11:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 515554304 bytes 2024-05-17T15:11:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 551536640 bytes 2024-05-17T15:11:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 552165376 bytes 2024-05-17T15:11:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 592536576 bytes 2024-05-17T15:11:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 649592832 bytes 2024-05-17T15:11:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 729053184 bytes 2024-05-17T15:11:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 804133888 bytes 2024-05-17T15:11:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 823220224 bytes 2024-05-17T15:11:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 876113920 bytes 2024-05-17T15:11:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 904175616 bytes 2024-05-17T15:11:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 981240832 bytes 2024-05-17T15:11:47.234 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1076826112 bytes 2024-05-17T15:11:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1120013312 bytes 2024-05-17T15:11:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1139610624 bytes 2024-05-17T15:11:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1172656128 bytes 2024-05-17T15:11:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1266535424 bytes 2024-05-17T15:11:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1284520960 bytes 2024-05-17T15:11:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1297790976 bytes 2024-05-17T15:11:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1313408000 bytes 2024-05-17T15:11:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1379900416 bytes 2024-05-17T15:11:48.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1444027392 bytes 2024-05-17T15:11:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1444642816 bytes 2024-05-17T15:11:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1481610240 bytes 2024-05-17T15:11:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1550734336 bytes 2024-05-17T15:11:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:11:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:11:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:11:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:11:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:11:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_40 2024-05-17T15:11:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-05-17T15:11:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_40 2024-05-17T15:11:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:11:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:11:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_40 128M 2024-05-17T15:11:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:48.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_40 2024-05-17T15:11:48.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:48.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:11:48.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:11:48.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:11:48.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:11:48.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_40 2024-05-17T15:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-05-17T15:11:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_40 2024-05-17T15:11:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:11:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-05-17T15:11:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:11:49.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-05-17T15:11:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_40': 2024-05-17T15:11:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:11:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:11:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:11:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 142f17080bb8 2024-05-17T15:11:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.142f17080bb8 2024-05-17T15:11:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:11:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:11:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:11:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:11:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:11:48 2024 2024-05-17T15:11:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:11:48 2024 2024-05-17T15:11:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:11:48 2024 2024-05-17T15:11:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:11:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:11:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 4abdb521-afae-4334-ad13-70815420eba8 2024-05-17T15:11:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:11:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:11:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_40 2024-05-17T15:11:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:11:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:11:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_40 2024-05-17T15:11:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:11:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=11 2024-05-17T15:11:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:11:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:11:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 29856768 bytes 2024-05-17T15:11:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 64638976 bytes 2024-05-17T15:11:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 95341568 bytes 2024-05-17T15:11:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 132950016 bytes 2024-05-17T15:11:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 217154560 bytes 2024-05-17T15:11:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 235907072 bytes 2024-05-17T15:11:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 301867008 bytes 2024-05-17T15:11:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 356189184 bytes 2024-05-17T15:11:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 457250816 bytes 2024-05-17T15:11:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 526798848 bytes 2024-05-17T15:11:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 569011200 bytes 2024-05-17T15:11:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 612353024 bytes 2024-05-17T15:11:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 644918272 bytes 2024-05-17T15:11:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 654879744 bytes 2024-05-17T15:11:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 715410432 bytes 2024-05-17T15:11:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 721539072 bytes 2024-05-17T15:11:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 761978880 bytes 2024-05-17T15:11:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 829829120 bytes 2024-05-17T15:11:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 871046144 bytes 2024-05-17T15:11:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 964222976 bytes 2024-05-17T15:11:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 966237184 bytes 2024-05-17T15:11:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1002289152 bytes 2024-05-17T15:11:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1142419456 bytes 2024-05-17T15:11:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1196967936 bytes 2024-05-17T15:11:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1229310976 bytes 2024-05-17T15:11:53.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1244429312 bytes 2024-05-17T15:11:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1297551360 bytes 2024-05-17T15:11:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1357238272 bytes 2024-05-17T15:11:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1381912576 bytes 2024-05-17T15:11:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1410830336 bytes 2024-05-17T15:11:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1434413056 bytes 2024-05-17T15:11:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1459544064 bytes 2024-05-17T15:11:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1533669376 bytes 2024-05-17T15:11:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1551475712 bytes 2024-05-17T15:11:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1562163200 bytes 2024-05-17T15:11:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1638870016 bytes 2024-05-17T15:11:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1656870912 bytes 2024-05-17T15:11:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1731020800 bytes 2024-05-17T15:11:55.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1771216896 bytes 2024-05-17T15:11:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1801693184 bytes 2024-05-17T15:11:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1872780288 bytes 2024-05-17T15:11:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1893736448 bytes 2024-05-17T15:11:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1899294720 bytes 2024-05-17T15:11:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1917108224 bytes 2024-05-17T15:11:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2027997184 bytes 2024-05-17T15:11:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2060030976 bytes 2024-05-17T15:11:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2090945536 bytes 2024-05-17T15:11:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2109221888 bytes 2024-05-17T15:11:57.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2151287808 bytes 2024-05-17T15:11:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2271929344 bytes 2024-05-17T15:11:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2312351744 bytes 2024-05-17T15:11:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2387196928 bytes 2024-05-17T15:11:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2419375104 bytes 2024-05-17T15:11:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2481196032 bytes 2024-05-17T15:11:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2522984448 bytes 2024-05-17T15:11:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2597270528 bytes 2024-05-17T15:11:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2625860608 bytes 2024-05-17T15:11:59.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2678977536 bytes 2024-05-17T15:11:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2685063168 bytes 2024-05-17T15:11:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2736358400 bytes 2024-05-17T15:11:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2803214336 bytes 2024-05-17T15:11:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2806935552 bytes 2024-05-17T15:11:59.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2826296320 bytes 2024-05-17T15:11:59.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2898396160 bytes 2024-05-17T15:11:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2931439616 bytes 2024-05-17T15:12:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2933485568 bytes 2024-05-17T15:12:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2965919744 bytes 2024-05-17T15:12:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 3014420480 bytes 2024-05-17T15:12:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 3028121600 bytes 2024-05-17T15:12:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 3041816576 bytes 2024-05-17T15:12:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3070330880 bytes 2024-05-17T15:12:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3116422144 bytes 2024-05-17T15:12:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3153441792 bytes 2024-05-17T15:12:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3160622080 bytes 2024-05-17T15:12:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:12:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:12:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:12:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:12:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:12:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_41 2024-05-17T15:12:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-05-17T15:12:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_41 2024-05-17T15:12:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:12:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:12:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_41 128M 2024-05-17T15:12:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_41 2024-05-17T15:12:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:12:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:12:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:12:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_41 2024-05-17T15:12:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:12:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:12:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:12:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-05-17T15:12:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_41 2024-05-17T15:12:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:12:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-05-17T15:12:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:12:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-05-17T15:12:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_41': 2024-05-17T15:12:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:12:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:12:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:12:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 143b7cb38b0b 2024-05-17T15:12:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.143b7cb38b0b 2024-05-17T15:12:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:12:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:12:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:12:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:12:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:12:01 2024 2024-05-17T15:12:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:12:01 2024 2024-05-17T15:12:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:12:01 2024 2024-05-17T15:12:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:12:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:12:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 69d8440b-93d1-40e8-b675-4ab5105af9bd 2024-05-17T15:12:01.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:12:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:12:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_41 2024-05-17T15:12:01.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_41 2024-05-17T15:12:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:12:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=7 2024-05-17T15:12:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:12:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:12:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 4609024 bytes 2024-05-17T15:12:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 35979264 bytes 2024-05-17T15:12:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 108573696 bytes 2024-05-17T15:12:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 134273024 bytes 2024-05-17T15:12:02.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 141690880 bytes 2024-05-17T15:12:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 235990016 bytes 2024-05-17T15:12:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 346902528 bytes 2024-05-17T15:12:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 432879616 bytes 2024-05-17T15:12:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 461689856 bytes 2024-05-17T15:12:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 522316800 bytes 2024-05-17T15:12:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 636481536 bytes 2024-05-17T15:12:03.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 663431168 bytes 2024-05-17T15:12:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 684110848 bytes 2024-05-17T15:12:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 744896512 bytes 2024-05-17T15:12:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 799984640 bytes 2024-05-17T15:12:04.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 843923456 bytes 2024-05-17T15:12:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 854285312 bytes 2024-05-17T15:12:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 865666048 bytes 2024-05-17T15:12:04.449 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 900678656 bytes 2024-05-17T15:12:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 960228352 bytes 2024-05-17T15:12:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 1036741632 bytes 2024-05-17T15:12:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1111382016 bytes 2024-05-17T15:12:05.166 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1179781120 bytes 2024-05-17T15:12:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1193292800 bytes 2024-05-17T15:12:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1257873408 bytes 2024-05-17T15:12:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1292584960 bytes 2024-05-17T15:12:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1293203456 bytes 2024-05-17T15:12:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1301498880 bytes 2024-05-17T15:12:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1346853888 bytes 2024-05-17T15:12:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1417157632 bytes 2024-05-17T15:12:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1548342272 bytes 2024-05-17T15:12:06.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1601137664 bytes 2024-05-17T15:12:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1649982464 bytes 2024-05-17T15:12:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1691713536 bytes 2024-05-17T15:12:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1745986560 bytes 2024-05-17T15:12:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1808815104 bytes 2024-05-17T15:12:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1841255424 bytes 2024-05-17T15:12:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1878819840 bytes 2024-05-17T15:12:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1898575872 bytes 2024-05-17T15:12:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1946498048 bytes 2024-05-17T15:12:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1986794496 bytes 2024-05-17T15:12:07.826 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1987414016 bytes 2024-05-17T15:12:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 2055499776 bytes 2024-05-17T15:12:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 2084830208 bytes 2024-05-17T15:12:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2122489856 bytes 2024-05-17T15:12:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2159860736 bytes 2024-05-17T15:12:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2245475328 bytes 2024-05-17T15:12:08.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:12:08.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:12:08.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:12:08.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:12:08.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:12:08.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_42 2024-05-17T15:12:08.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-05-17T15:12:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_42 2024-05-17T15:12:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:12:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:12:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_42 128M 2024-05-17T15:12:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_42 2024-05-17T15:12:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:12:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:12:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:12:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_42 2024-05-17T15:12:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:12:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:12:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:12:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-05-17T15:12:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_42 2024-05-17T15:12:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:12:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-05-17T15:12:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:12:09.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-05-17T15:12:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_42': 2024-05-17T15:12:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:12:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:12:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:12:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 144768a04432 2024-05-17T15:12:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.144768a04432 2024-05-17T15:12:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:12:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:12:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:12:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:12:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:12:08 2024 2024-05-17T15:12:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:12:08 2024 2024-05-17T15:12:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:12:08 2024 2024-05-17T15:12:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:12:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:12:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 46ae8559-1830-4c3d-adfc-c8f96345efad 2024-05-17T15:12:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:12:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:12:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_42 2024-05-17T15:12:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_42 2024-05-17T15:12:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:12:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=6 2024-05-17T15:12:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:12:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:12:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 33471488 bytes 2024-05-17T15:12:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 34933760 bytes 2024-05-17T15:12:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 102009856 bytes 2024-05-17T15:12:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 144121856 bytes 2024-05-17T15:12:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 219139072 bytes 2024-05-17T15:12:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 321984512 bytes 2024-05-17T15:12:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 332604416 bytes 2024-05-17T15:12:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 343936000 bytes 2024-05-17T15:12:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 384293888 bytes 2024-05-17T15:12:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 473978880 bytes 2024-05-17T15:12:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 499643392 bytes 2024-05-17T15:12:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 535491584 bytes 2024-05-17T15:12:11.519 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 585482240 bytes 2024-05-17T15:12:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 648610816 bytes 2024-05-17T15:12:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 698156032 bytes 2024-05-17T15:12:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 742604800 bytes 2024-05-17T15:12:12.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 807283712 bytes 2024-05-17T15:12:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 892924928 bytes 2024-05-17T15:12:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 947814400 bytes 2024-05-17T15:12:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 995831808 bytes 2024-05-17T15:12:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 1075649536 bytes 2024-05-17T15:12:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1138596864 bytes 2024-05-17T15:12:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1223772160 bytes 2024-05-17T15:12:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1260018688 bytes 2024-05-17T15:12:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1303541760 bytes 2024-05-17T15:12:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1331547136 bytes 2024-05-17T15:12:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1375342592 bytes 2024-05-17T15:12:13.805 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1401341952 bytes 2024-05-17T15:12:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1474324480 bytes 2024-05-17T15:12:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1517352960 bytes 2024-05-17T15:12:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1607442432 bytes 2024-05-17T15:12:14.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1685217280 bytes 2024-05-17T15:12:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1739106304 bytes 2024-05-17T15:12:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1817766912 bytes 2024-05-17T15:12:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1853168640 bytes 2024-05-17T15:12:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1924063232 bytes 2024-05-17T15:12:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1943183360 bytes 2024-05-17T15:12:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1944155136 bytes 2024-05-17T15:12:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1987078144 bytes 2024-05-17T15:12:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:12:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:12:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:12:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:12:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:12:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_43 2024-05-17T15:12:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-05-17T15:12:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_43 2024-05-17T15:12:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:12:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:12:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_43 128M 2024-05-17T15:12:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_43 2024-05-17T15:12:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:12:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:12:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:12:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_43 2024-05-17T15:12:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:12:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:12:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:12:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-05-17T15:12:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_43 2024-05-17T15:12:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:12:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-05-17T15:12:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:12:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-05-17T15:12:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_43': 2024-05-17T15:12:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:12:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:12:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:12:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 145352e66c54 2024-05-17T15:12:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.145352e66c54 2024-05-17T15:12:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:12:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:12:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:12:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:12:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:12:15 2024 2024-05-17T15:12:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:12:15 2024 2024-05-17T15:12:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:12:15 2024 2024-05-17T15:12:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:12:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:12:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 414f7604-42d0-4537-bea3-bfcfbe66e210 2024-05-17T15:12:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:12:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:12:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_43 2024-05-17T15:12:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_43 2024-05-17T15:12:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:12:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=5 2024-05-17T15:12:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:12:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:12:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 7607296 bytes 2024-05-17T15:12:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 84600832 bytes 2024-05-17T15:12:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 130314240 bytes 2024-05-17T15:12:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 173626368 bytes 2024-05-17T15:12:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 256596992 bytes 2024-05-17T15:12:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 287144960 bytes 2024-05-17T15:12:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 316479488 bytes 2024-05-17T15:12:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 374231040 bytes 2024-05-17T15:12:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 433758208 bytes 2024-05-17T15:12:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 435404800 bytes 2024-05-17T15:12:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 484077568 bytes 2024-05-17T15:12:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 513796096 bytes 2024-05-17T15:12:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 570975232 bytes 2024-05-17T15:12:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 593071104 bytes 2024-05-17T15:12:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 603877376 bytes 2024-05-17T15:12:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 630093824 bytes 2024-05-17T15:12:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 668541952 bytes 2024-05-17T15:12:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 744708096 bytes 2024-05-17T15:12:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 786880512 bytes 2024-05-17T15:12:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 864561152 bytes 2024-05-17T15:12:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 905629696 bytes 2024-05-17T15:12:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 920680448 bytes 2024-05-17T15:12:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 948225024 bytes 2024-05-17T15:12:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 979635200 bytes 2024-05-17T15:12:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1042375680 bytes 2024-05-17T15:12:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1085940736 bytes 2024-05-17T15:12:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1141288960 bytes 2024-05-17T15:12:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1231171584 bytes 2024-05-17T15:12:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1262869504 bytes 2024-05-17T15:12:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1304129536 bytes 2024-05-17T15:12:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1358527488 bytes 2024-05-17T15:12:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1451031552 bytes 2024-05-17T15:12:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1502200832 bytes 2024-05-17T15:12:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:12:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:12:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:12:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:12:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:12:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_44 2024-05-17T15:12:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-05-17T15:12:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_44 2024-05-17T15:12:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:12:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:12:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_44 128M 2024-05-17T15:12:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_44 2024-05-17T15:12:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:12:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:12:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:12:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_44 2024-05-17T15:12:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:12:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:12:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:12:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-05-17T15:12:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_44 2024-05-17T15:12:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:12:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-05-17T15:12:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:12:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-05-17T15:12:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_44': 2024-05-17T15:12:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:12:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:12:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:12:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 145f188ade6a 2024-05-17T15:12:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.145f188ade6a 2024-05-17T15:12:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:12:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:12:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:12:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:12:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:12:21 2024 2024-05-17T15:12:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:12:21 2024 2024-05-17T15:12:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:12:21 2024 2024-05-17T15:12:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:12:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:12:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 42d7dd77-f5e1-4c19-a790-e2073f5b67ee 2024-05-17T15:12:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:12:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:12:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_44 2024-05-17T15:12:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_44 2024-05-17T15:12:22.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:12:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=7 2024-05-17T15:12:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:12:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:12:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 36771840 bytes 2024-05-17T15:12:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 66736128 bytes 2024-05-17T15:12:23.192 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 149138432 bytes 2024-05-17T15:12:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 204778496 bytes 2024-05-17T15:12:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 253575168 bytes 2024-05-17T15:12:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 294788096 bytes 2024-05-17T15:12:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 315888640 bytes 2024-05-17T15:12:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 340213760 bytes 2024-05-17T15:12:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 371973120 bytes 2024-05-17T15:12:24.271 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 372597760 bytes 2024-05-17T15:12:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 456177664 bytes 2024-05-17T15:12:24.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 456781824 bytes 2024-05-17T15:12:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 469925888 bytes 2024-05-17T15:12:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 496101376 bytes 2024-05-17T15:12:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 538194944 bytes 2024-05-17T15:12:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 576717824 bytes 2024-05-17T15:12:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 624292864 bytes 2024-05-17T15:12:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 651713536 bytes 2024-05-17T15:12:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 730306560 bytes 2024-05-17T15:12:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 773765120 bytes 2024-05-17T15:12:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 817568768 bytes 2024-05-17T15:12:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 818188288 bytes 2024-05-17T15:12:25.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 848590848 bytes 2024-05-17T15:12:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 896694272 bytes 2024-05-17T15:12:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 947066880 bytes 2024-05-17T15:12:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1039241216 bytes 2024-05-17T15:12:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1153014784 bytes 2024-05-17T15:12:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1187704832 bytes 2024-05-17T15:12:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1198296064 bytes 2024-05-17T15:12:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1251601408 bytes 2024-05-17T15:12:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1274716160 bytes 2024-05-17T15:12:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1314850816 bytes 2024-05-17T15:12:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1376867328 bytes 2024-05-17T15:12:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1476606976 bytes 2024-05-17T15:12:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1496014848 bytes 2024-05-17T15:12:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1545055232 bytes 2024-05-17T15:12:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1626737664 bytes 2024-05-17T15:12:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1639494656 bytes 2024-05-17T15:12:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1686891520 bytes 2024-05-17T15:12:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1709844480 bytes 2024-05-17T15:12:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1749785600 bytes 2024-05-17T15:12:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1787850752 bytes 2024-05-17T15:12:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1808317440 bytes 2024-05-17T15:12:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1864346624 bytes 2024-05-17T15:12:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1907240960 bytes 2024-05-17T15:12:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1954989056 bytes 2024-05-17T15:12:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2022541312 bytes 2024-05-17T15:12:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2023151616 bytes 2024-05-17T15:12:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2026238976 bytes 2024-05-17T15:12:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:12:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:12:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:12:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:12:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:12:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_45 2024-05-17T15:12:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-05-17T15:12:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_45 2024-05-17T15:12:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:12:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:12:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_45 128M 2024-05-17T15:12:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_45 2024-05-17T15:12:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:12:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:12:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:12:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_45 2024-05-17T15:12:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:12:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:12:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:12:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-05-17T15:12:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_45 2024-05-17T15:12:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:12:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-05-17T15:12:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:12:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-05-17T15:12:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_45': 2024-05-17T15:12:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:12:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:12:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:12:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 146b5f88ef9b 2024-05-17T15:12:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.146b5f88ef9b 2024-05-17T15:12:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:12:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:12:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:12:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:12:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:12:29 2024 2024-05-17T15:12:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:12:29 2024 2024-05-17T15:12:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:12:29 2024 2024-05-17T15:12:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:12:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:12:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 729f0874-9d81-4812-83cf-ae9935d33036 2024-05-17T15:12:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:12:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:12:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_45 2024-05-17T15:12:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_45 2024-05-17T15:12:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:12:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=14 2024-05-17T15:12:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:12:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:12:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 103058432 bytes 2024-05-17T15:12:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 127551488 bytes 2024-05-17T15:12:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 162259968 bytes 2024-05-17T15:12:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 201078784 bytes 2024-05-17T15:12:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 213182464 bytes 2024-05-17T15:12:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 235315200 bytes 2024-05-17T15:12:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 277694464 bytes 2024-05-17T15:12:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 293489664 bytes 2024-05-17T15:12:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 296970240 bytes 2024-05-17T15:12:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 315154432 bytes 2024-05-17T15:12:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 364531712 bytes 2024-05-17T15:12:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 397570048 bytes 2024-05-17T15:12:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 443649024 bytes 2024-05-17T15:12:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 444266496 bytes 2024-05-17T15:12:31.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 472300544 bytes 2024-05-17T15:12:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 501331968 bytes 2024-05-17T15:12:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 551628800 bytes 2024-05-17T15:12:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 612120576 bytes 2024-05-17T15:12:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 650579968 bytes 2024-05-17T15:12:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 705831936 bytes 2024-05-17T15:12:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 743349248 bytes 2024-05-17T15:12:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 810452992 bytes 2024-05-17T15:12:33.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 853869568 bytes 2024-05-17T15:12:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 866132992 bytes 2024-05-17T15:12:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 866780160 bytes 2024-05-17T15:12:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 938613760 bytes 2024-05-17T15:12:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 968758272 bytes 2024-05-17T15:12:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1040739328 bytes 2024-05-17T15:12:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1069769728 bytes 2024-05-17T15:12:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1103117312 bytes 2024-05-17T15:12:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1127690240 bytes 2024-05-17T15:12:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1170906112 bytes 2024-05-17T15:12:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1220635648 bytes 2024-05-17T15:12:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1270662144 bytes 2024-05-17T15:12:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1321009152 bytes 2024-05-17T15:12:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1356312576 bytes 2024-05-17T15:12:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1396356096 bytes 2024-05-17T15:12:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1423422464 bytes 2024-05-17T15:12:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1442700288 bytes 2024-05-17T15:12:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1477177344 bytes 2024-05-17T15:12:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1557828608 bytes 2024-05-17T15:12:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1620472832 bytes 2024-05-17T15:12:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1684343808 bytes 2024-05-17T15:12:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1744754688 bytes 2024-05-17T15:12:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1784324096 bytes 2024-05-17T15:12:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1810077696 bytes 2024-05-17T15:12:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1850323968 bytes 2024-05-17T15:12:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 1850962944 bytes 2024-05-17T15:12:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 1890678784 bytes 2024-05-17T15:12:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 1906651136 bytes 2024-05-17T15:12:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 1913812992 bytes 2024-05-17T15:12:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 1951863808 bytes 2024-05-17T15:12:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 1995848704 bytes 2024-05-17T15:12:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2017179648 bytes 2024-05-17T15:12:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2110438400 bytes 2024-05-17T15:12:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2204585984 bytes 2024-05-17T15:12:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2229676032 bytes 2024-05-17T15:12:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2302547968 bytes 2024-05-17T15:12:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2339400704 bytes 2024-05-17T15:12:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2354903040 bytes 2024-05-17T15:12:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2401659904 bytes 2024-05-17T15:12:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2462280704 bytes 2024-05-17T15:12:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2481003520 bytes 2024-05-17T15:12:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2502622208 bytes 2024-05-17T15:12:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2539564032 bytes 2024-05-17T15:12:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2611954688 bytes 2024-05-17T15:12:39.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2671504384 bytes 2024-05-17T15:12:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2771562496 bytes 2024-05-17T15:12:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2804550656 bytes 2024-05-17T15:12:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 2820116480 bytes 2024-05-17T15:12:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 2846808064 bytes 2024-05-17T15:12:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 2869524480 bytes 2024-05-17T15:12:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 2996419584 bytes 2024-05-17T15:12:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3072989184 bytes 2024-05-17T15:12:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3110812672 bytes 2024-05-17T15:12:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3111410688 bytes 2024-05-17T15:12:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3146042368 bytes 2024-05-17T15:12:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3228680192 bytes 2024-05-17T15:12:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3257784320 bytes 2024-05-17T15:12:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3332636672 bytes 2024-05-17T15:12:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3391208448 bytes 2024-05-17T15:12:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3438863360 bytes 2024-05-17T15:12:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3494417408 bytes 2024-05-17T15:12:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3550923776 bytes 2024-05-17T15:12:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3592432640 bytes 2024-05-17T15:12:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3619049472 bytes 2024-05-17T15:12:42.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3644167168 bytes 2024-05-17T15:12:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3714201600 bytes 2024-05-17T15:12:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3767980032 bytes 2024-05-17T15:12:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3817528320 bytes 2024-05-17T15:12:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 3846407168 bytes 2024-05-17T15:12:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9200 IOs, 3873987584 bytes 2024-05-17T15:12:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9300 IOs, 3917030400 bytes 2024-05-17T15:12:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9400 IOs, 3957931008 bytes 2024-05-17T15:12:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:12:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:12:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:12:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:12:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:12:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_46 2024-05-17T15:12:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-05-17T15:12:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_46 2024-05-17T15:12:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:12:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:12:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_46 128M 2024-05-17T15:12:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_46 2024-05-17T15:12:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:12:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:12:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:12:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:43.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_46 2024-05-17T15:12:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:12:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:12:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:12:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-05-17T15:12:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_46 2024-05-17T15:12:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:12:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-05-17T15:12:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:12:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-05-17T15:12:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_46': 2024-05-17T15:12:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:12:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:12:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:12:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1477264d4ae9 2024-05-17T15:12:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1477264d4ae9 2024-05-17T15:12:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:12:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:12:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:12:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:12:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:12:43 2024 2024-05-17T15:12:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:12:43 2024 2024-05-17T15:12:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:12:43 2024 2024-05-17T15:12:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:12:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:12:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 36a3913d-378a-412a-ad5b-9c269860173e 2024-05-17T15:12:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:12:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:12:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_46 2024-05-17T15:12:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_46 2024-05-17T15:12:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:12:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=5 2024-05-17T15:12:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:12:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:12:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 20580352 bytes 2024-05-17T15:12:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 42525696 bytes 2024-05-17T15:12:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 74574848 bytes 2024-05-17T15:12:45.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 111592448 bytes 2024-05-17T15:12:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 117282816 bytes 2024-05-17T15:12:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 158584832 bytes 2024-05-17T15:12:45.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 202605568 bytes 2024-05-17T15:12:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 253292544 bytes 2024-05-17T15:12:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 309578752 bytes 2024-05-17T15:12:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 400094208 bytes 2024-05-17T15:12:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 429368320 bytes 2024-05-17T15:12:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 475633664 bytes 2024-05-17T15:12:46.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 524933120 bytes 2024-05-17T15:12:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 546849792 bytes 2024-05-17T15:12:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 591586304 bytes 2024-05-17T15:12:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 708278272 bytes 2024-05-17T15:12:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 748913664 bytes 2024-05-17T15:12:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 821994496 bytes 2024-05-17T15:12:47.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 873448448 bytes 2024-05-17T15:12:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 918665216 bytes 2024-05-17T15:12:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 947119104 bytes 2024-05-17T15:12:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 957558784 bytes 2024-05-17T15:12:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 998812672 bytes 2024-05-17T15:12:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1086535680 bytes 2024-05-17T15:12:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1097359360 bytes 2024-05-17T15:12:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1137651712 bytes 2024-05-17T15:12:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1186774016 bytes 2024-05-17T15:12:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1241884672 bytes 2024-05-17T15:12:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1287763968 bytes 2024-05-17T15:12:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1328681984 bytes 2024-05-17T15:12:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1359838208 bytes 2024-05-17T15:12:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1408558080 bytes 2024-05-17T15:12:49.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1456165888 bytes 2024-05-17T15:12:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1495713792 bytes 2024-05-17T15:12:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:12:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:12:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:12:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:12:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:12:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_47 2024-05-17T15:12:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-05-17T15:12:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_47 2024-05-17T15:12:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:12:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:12:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_47 128M 2024-05-17T15:12:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_47 2024-05-17T15:12:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:12:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:12:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:12:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_47 2024-05-17T15:12:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:12:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:12:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:12:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-05-17T15:12:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_47 2024-05-17T15:12:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:12:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-05-17T15:12:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:12:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-05-17T15:12:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_47': 2024-05-17T15:12:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:12:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:12:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:12:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1483945d3dfe 2024-05-17T15:12:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1483945d3dfe 2024-05-17T15:12:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:12:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:12:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:12:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:12:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:12:49 2024 2024-05-17T15:12:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:12:49 2024 2024-05-17T15:12:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:12:49 2024 2024-05-17T15:12:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:12:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:12:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 84e9e05b-5031-4201-93ab-915cf8a3c889 2024-05-17T15:12:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:12:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:12:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_47 2024-05-17T15:12:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_47 2024-05-17T15:12:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:12:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=9 2024-05-17T15:12:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:12:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:12:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 106316800 bytes 2024-05-17T15:12:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 116529152 bytes 2024-05-17T15:12:51.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 179326976 bytes 2024-05-17T15:12:51.325 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 229720064 bytes 2024-05-17T15:12:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 262691840 bytes 2024-05-17T15:12:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 263319552 bytes 2024-05-17T15:12:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 323509248 bytes 2024-05-17T15:12:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 394746880 bytes 2024-05-17T15:12:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 420772864 bytes 2024-05-17T15:12:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 447258624 bytes 2024-05-17T15:12:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 447879168 bytes 2024-05-17T15:12:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 500262912 bytes 2024-05-17T15:12:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 528598016 bytes 2024-05-17T15:12:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 544799744 bytes 2024-05-17T15:12:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 580645888 bytes 2024-05-17T15:12:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 629764096 bytes 2024-05-17T15:12:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 669294592 bytes 2024-05-17T15:12:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 684530688 bytes 2024-05-17T15:12:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 739814400 bytes 2024-05-17T15:12:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 809985024 bytes 2024-05-17T15:12:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 827333632 bytes 2024-05-17T15:12:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 827945984 bytes 2024-05-17T15:12:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 870919168 bytes 2024-05-17T15:12:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 933019648 bytes 2024-05-17T15:12:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 993593344 bytes 2024-05-17T15:12:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1053528064 bytes 2024-05-17T15:12:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1089318912 bytes 2024-05-17T15:12:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1184902144 bytes 2024-05-17T15:12:54.696 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1192665088 bytes 2024-05-17T15:12:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1211670528 bytes 2024-05-17T15:12:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1258689536 bytes 2024-05-17T15:12:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1306698752 bytes 2024-05-17T15:12:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1321567232 bytes 2024-05-17T15:12:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1337822208 bytes 2024-05-17T15:12:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1355796480 bytes 2024-05-17T15:12:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1390929920 bytes 2024-05-17T15:12:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1460793344 bytes 2024-05-17T15:12:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1495421952 bytes 2024-05-17T15:12:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1543980032 bytes 2024-05-17T15:12:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1608005632 bytes 2024-05-17T15:12:56.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1631608832 bytes 2024-05-17T15:12:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1667448832 bytes 2024-05-17T15:12:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1728550912 bytes 2024-05-17T15:12:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1819274240 bytes 2024-05-17T15:12:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1846361088 bytes 2024-05-17T15:12:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1923030016 bytes 2024-05-17T15:12:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1965362176 bytes 2024-05-17T15:12:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 1995091968 bytes 2024-05-17T15:12:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2016649216 bytes 2024-05-17T15:12:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2050069504 bytes 2024-05-17T15:12:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2060090368 bytes 2024-05-17T15:12:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2123656192 bytes 2024-05-17T15:12:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2153733120 bytes 2024-05-17T15:12:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2168521728 bytes 2024-05-17T15:12:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2172985344 bytes 2024-05-17T15:12:58.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2210842624 bytes 2024-05-17T15:12:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2233390080 bytes 2024-05-17T15:12:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2287450112 bytes 2024-05-17T15:12:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2312147968 bytes 2024-05-17T15:12:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2342980608 bytes 2024-05-17T15:12:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2441024512 bytes 2024-05-17T15:12:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2522432512 bytes 2024-05-17T15:12:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2540951552 bytes 2024-05-17T15:12:59.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2594314240 bytes 2024-05-17T15:12:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2614518784 bytes 2024-05-17T15:12:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:12:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:12:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:12:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:12:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:12:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_48 2024-05-17T15:12:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-05-17T15:12:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_48 2024-05-17T15:12:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:12:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:12:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_48 128M 2024-05-17T15:12:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_48 2024-05-17T15:12:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:12:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:12:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:12:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:12:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_48 2024-05-17T15:12:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:12:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:12:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:12:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-05-17T15:12:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:12:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:12:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_48 2024-05-17T15:12:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:12:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-05-17T15:13:00.577 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:13:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-05-17T15:13:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_48': 2024-05-17T15:13:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:13:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:13:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:13:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 148fa6de30a5 2024-05-17T15:13:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.148fa6de30a5 2024-05-17T15:13:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:13:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:13:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:13:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:13:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:12:59 2024 2024-05-17T15:13:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:12:59 2024 2024-05-17T15:13:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:12:59 2024 2024-05-17T15:13:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:13:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:13:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 84523fc1-2802-4868-9b68-13d10e9be7e2 2024-05-17T15:13:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:13:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:13:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_48 2024-05-17T15:13:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:13:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:13:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_48 2024-05-17T15:13:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:13:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=11 2024-05-17T15:13:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:13:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:13:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 54749184 bytes 2024-05-17T15:13:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 106696704 bytes 2024-05-17T15:13:01.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 161476608 bytes 2024-05-17T15:13:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 208051200 bytes 2024-05-17T15:13:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 208660480 bytes 2024-05-17T15:13:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 209286144 bytes 2024-05-17T15:13:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 249616384 bytes 2024-05-17T15:13:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 311275520 bytes 2024-05-17T15:13:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 402358272 bytes 2024-05-17T15:13:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 441714688 bytes 2024-05-17T15:13:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 450810880 bytes 2024-05-17T15:13:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 470856704 bytes 2024-05-17T15:13:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 522155008 bytes 2024-05-17T15:13:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 612677632 bytes 2024-05-17T15:13:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 716318720 bytes 2024-05-17T15:13:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 789497856 bytes 2024-05-17T15:13:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 889198592 bytes 2024-05-17T15:13:03.430 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 938499072 bytes 2024-05-17T15:13:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 997057536 bytes 2024-05-17T15:13:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 1010169856 bytes 2024-05-17T15:13:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 1063625728 bytes 2024-05-17T15:13:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1069284352 bytes 2024-05-17T15:13:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1102911488 bytes 2024-05-17T15:13:04.164 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1115854848 bytes 2024-05-17T15:13:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1134722048 bytes 2024-05-17T15:13:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1194164224 bytes 2024-05-17T15:13:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1244989440 bytes 2024-05-17T15:13:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1306582016 bytes 2024-05-17T15:13:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1337895936 bytes 2024-05-17T15:13:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1386307584 bytes 2024-05-17T15:13:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1454820352 bytes 2024-05-17T15:13:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1496393728 bytes 2024-05-17T15:13:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1538988032 bytes 2024-05-17T15:13:05.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1575009280 bytes 2024-05-17T15:13:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1596390400 bytes 2024-05-17T15:13:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1612861440 bytes 2024-05-17T15:13:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1658166272 bytes 2024-05-17T15:13:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1673075712 bytes 2024-05-17T15:13:06.249 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:2024-05-17T15:13:06.246+0000 7f9387ac4640 -1 bdev(0x5569ad194e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-05-17T15:13:06.250 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3735-gee59d86a/rpm/el9/BUILD/ceph-19.0.0-3735-gee59d86a/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f9387ac4640 time 2024-05-17T15:13:06.248231+0000 2024-05-17T15:13:06.250 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3735-gee59d86a/rpm/el9/BUILD/ceph-19.0.0-3735-gee59d86a/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-05-17T15:13:06.250 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev) 2024-05-17T15:13:06.250 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5569ab4b48e6] 2024-05-17T15:13:06.250 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x5569abe8d2f0] 2024-05-17T15:13:06.251 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 3: ceph-osd(+0xdcb461) [0x5569abe8d461] 2024-05-17T15:13:06.251 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 4: /lib64/libc.so.6(+0x89c02) [0x7f9394a89c02] 2024-05-17T15:13:06.251 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 5: /lib64/libc.so.6(+0x10ec40) [0x7f9394b0ec40] 2024-05-17T15:13:06.251 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:2024-05-17T15:13:06.247+0000 7f9387ac4640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3735-gee59d86a/rpm/el9/BUILD/ceph-19.0.0-3735-gee59d86a/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f9387ac4640 time 2024-05-17T15:13:06.248231+0000 2024-05-17T15:13:06.251 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3735-gee59d86a/rpm/el9/BUILD/ceph-19.0.0-3735-gee59d86a/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-05-17T15:13:06.251 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2024-05-17T15:13:06.251 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev) 2024-05-17T15:13:06.251 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5569ab4b48e6] 2024-05-17T15:13:06.251 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x5569abe8d2f0] 2024-05-17T15:13:06.251 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 3: ceph-osd(+0xdcb461) [0x5569abe8d461] 2024-05-17T15:13:06.251 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 4: /lib64/libc.so.6(+0x89c02) [0x7f9394a89c02] 2024-05-17T15:13:06.252 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 5: /lib64/libc.so.6(+0x10ec40) [0x7f9394b0ec40] 2024-05-17T15:13:06.252 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2024-05-17T15:13:06.252 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:*** Caught signal (Aborted) ** 2024-05-17T15:13:06.252 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: in thread 7f9387ac4640 thread_name:bstore_aio 2024-05-17T15:13:06.252 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev) 2024-05-17T15:13:06.252 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9394a3e6f0] 2024-05-17T15:13:06.252 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9394a8b94c] 2024-05-17T15:13:06.252 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 3: raise() 2024-05-17T15:13:06.252 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 4: abort() 2024-05-17T15:13:06.253 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x5569ab4b49a7] 2024-05-17T15:13:06.253 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x5569abe8d2f0] 2024-05-17T15:13:06.253 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 7: ceph-osd(+0xdcb461) [0x5569abe8d461] 2024-05-17T15:13:06.253 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 8: /lib64/libc.so.6(+0x89c02) [0x7f9394a89c02] 2024-05-17T15:13:06.253 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 9: /lib64/libc.so.6(+0x10ec40) [0x7f9394b0ec40] 2024-05-17T15:13:06.253 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:2024-05-17T15:13:06.248+0000 7f9387ac4640 -1 *** Caught signal (Aborted) ** 2024-05-17T15:13:06.253 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: in thread 7f9387ac4640 thread_name:bstore_aio 2024-05-17T15:13:06.253 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2024-05-17T15:13:06.253 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev) 2024-05-17T15:13:06.253 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9394a3e6f0] 2024-05-17T15:13:06.253 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9394a8b94c] 2024-05-17T15:13:06.253 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 3: raise() 2024-05-17T15:13:06.253 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 4: abort() 2024-05-17T15:13:06.254 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x5569ab4b49a7] 2024-05-17T15:13:06.254 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x5569abe8d2f0] 2024-05-17T15:13:06.254 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 7: ceph-osd(+0xdcb461) [0x5569abe8d461] 2024-05-17T15:13:06.254 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 8: /lib64/libc.so.6(+0x89c02) [0x7f9394a89c02] 2024-05-17T15:13:06.254 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 9: /lib64/libc.so.6(+0x10ec40) [0x7f9394b0ec40] 2024-05-17T15:13:06.254 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-17T15:13:06.254 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2024-05-17T15:13:06.268 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: -37> 2024-05-17T15:13:06.246+0000 7f9387ac4640 -1 bdev(0x5569ad194e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-05-17T15:13:06.268 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: -3> 2024-05-17T15:13:06.247+0000 7f9387ac4640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3735-gee59d86a/rpm/el9/BUILD/ceph-19.0.0-3735-gee59d86a/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f9387ac4640 time 2024-05-17T15:13:06.248231+0000 2024-05-17T15:13:06.268 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3735-gee59d86a/rpm/el9/BUILD/ceph-19.0.0-3735-gee59d86a/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-05-17T15:13:06.269 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2024-05-17T15:13:06.269 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev) 2024-05-17T15:13:06.269 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5569ab4b48e6] 2024-05-17T15:13:06.269 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x5569abe8d2f0] 2024-05-17T15:13:06.269 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 3: ceph-osd(+0xdcb461) [0x5569abe8d461] 2024-05-17T15:13:06.269 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 4: /lib64/libc.so.6(+0x89c02) [0x7f9394a89c02] 2024-05-17T15:13:06.269 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 5: /lib64/libc.so.6(+0x10ec40) [0x7f9394b0ec40] 2024-05-17T15:13:06.269 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2024-05-17T15:13:06.269 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 0> 2024-05-17T15:13:06.248+0000 7f9387ac4640 -1 *** Caught signal (Aborted) ** 2024-05-17T15:13:06.269 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: in thread 7f9387ac4640 thread_name:bstore_aio 2024-05-17T15:13:06.269 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2024-05-17T15:13:06.269 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev) 2024-05-17T15:13:06.269 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9394a3e6f0] 2024-05-17T15:13:06.269 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9394a8b94c] 2024-05-17T15:13:06.269 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 3: raise() 2024-05-17T15:13:06.270 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 4: abort() 2024-05-17T15:13:06.270 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x5569ab4b49a7] 2024-05-17T15:13:06.270 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x5569abe8d2f0] 2024-05-17T15:13:06.270 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 7: ceph-osd(+0xdcb461) [0x5569abe8d461] 2024-05-17T15:13:06.270 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 8: /lib64/libc.so.6(+0x89c02) [0x7f9394a89c02] 2024-05-17T15:13:06.270 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 9: /lib64/libc.so.6(+0x10ec40) [0x7f9394b0ec40] 2024-05-17T15:13:06.270 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-17T15:13:06.270 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2024-05-17T15:13:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1756436480 bytes 2024-05-17T15:13:06.279 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: -727> 2024-05-17T15:13:06.246+0000 7f9387ac4640 -1 bdev(0x5569ad194e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2024-05-17T15:13:06.279 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: -693> 2024-05-17T15:13:06.247+0000 7f9387ac4640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3735-gee59d86a/rpm/el9/BUILD/ceph-19.0.0-3735-gee59d86a/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f9387ac4640 time 2024-05-17T15:13:06.248231+0000 2024-05-17T15:13:06.279 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3735-gee59d86a/rpm/el9/BUILD/ceph-19.0.0-3735-gee59d86a/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-05-17T15:13:06.279 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2024-05-17T15:13:06.279 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev) 2024-05-17T15:13:06.279 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5569ab4b48e6] 2024-05-17T15:13:06.279 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x5569abe8d2f0] 2024-05-17T15:13:06.279 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 3: ceph-osd(+0xdcb461) [0x5569abe8d461] 2024-05-17T15:13:06.279 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 4: /lib64/libc.so.6(+0x89c02) [0x7f9394a89c02] 2024-05-17T15:13:06.279 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 5: /lib64/libc.so.6(+0x10ec40) [0x7f9394b0ec40] 2024-05-17T15:13:06.279 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2024-05-17T15:13:06.280 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: -690> 2024-05-17T15:13:06.248+0000 7f9387ac4640 -1 *** Caught signal (Aborted) ** 2024-05-17T15:13:06.280 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: in thread 7f9387ac4640 thread_name:bstore_aio 2024-05-17T15:13:06.280 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2024-05-17T15:13:06.280 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev) 2024-05-17T15:13:06.280 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9394a3e6f0] 2024-05-17T15:13:06.280 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9394a8b94c] 2024-05-17T15:13:06.280 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 3: raise() 2024-05-17T15:13:06.280 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 4: abort() 2024-05-17T15:13:06.280 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x5569ab4b49a7] 2024-05-17T15:13:06.280 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 6: (KernelDevice::_aio_thread()+0x11e0) [0x5569abe8d2f0] 2024-05-17T15:13:06.280 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 7: ceph-osd(+0xdcb461) [0x5569abe8d461] 2024-05-17T15:13:06.280 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 8: /lib64/libc.so.6(+0x89c02) [0x7f9394a89c02] 2024-05-17T15:13:06.280 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 9: /lib64/libc.so.6(+0x10ec40) [0x7f9394b0ec40] 2024-05-17T15:13:06.280 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-17T15:13:06.281 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr: 2024-05-17T15:13:06.517 INFO:tasks.ceph.cluster2.osd.0.smithi195.stderr:daemon-helper: command failed with exit status 5 2024-05-17T15:13:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1806912512 bytes 2024-05-17T15:13:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1862265856 bytes 2024-05-17T15:13:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1903033344 bytes 2024-05-17T15:13:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1909933056 bytes 2024-05-17T15:13:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1953015808 bytes 2024-05-17T15:13:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2034713600 bytes 2024-05-17T15:13:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2104093696 bytes 2024-05-17T15:13:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2123161600 bytes 2024-05-17T15:13:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2160412672 bytes 2024-05-17T15:13:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2161060864 bytes 2024-05-17T15:13:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2196023296 bytes 2024-05-17T15:13:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2196630528 bytes 2024-05-17T15:13:08.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2256653312 bytes 2024-05-17T15:13:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2303632384 bytes 2024-05-17T15:13:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2360838144 bytes 2024-05-17T15:13:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2405610496 bytes 2024-05-17T15:13:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2420958208 bytes 2024-05-17T15:13:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2448972800 bytes 2024-05-17T15:13:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2496855040 bytes 2024-05-17T15:13:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2515666944 bytes 2024-05-17T15:13:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2534088704 bytes 2024-05-17T15:13:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2569329664 bytes 2024-05-17T15:13:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2603447296 bytes 2024-05-17T15:13:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2630877184 bytes 2024-05-17T15:13:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2703971328 bytes 2024-05-17T15:13:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2718488576 bytes 2024-05-17T15:13:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2742807552 bytes 2024-05-17T15:13:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2799539200 bytes 2024-05-17T15:13:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2814755840 bytes 2024-05-17T15:13:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 2870639616 bytes 2024-05-17T15:13:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 2948688896 bytes 2024-05-17T15:13:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 2991078400 bytes 2024-05-17T15:13:10.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3008261120 bytes 2024-05-17T15:13:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3036526592 bytes 2024-05-17T15:13:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3081935872 bytes 2024-05-17T15:13:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3118772224 bytes 2024-05-17T15:13:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3258205184 bytes 2024-05-17T15:13:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3287326720 bytes 2024-05-17T15:13:11.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3287946240 bytes 2024-05-17T15:13:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3318731776 bytes 2024-05-17T15:13:11.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3334769664 bytes 2024-05-17T15:13:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3367047168 bytes 2024-05-17T15:13:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3419476992 bytes 2024-05-17T15:13:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:13:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:13:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:13:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:13:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:13:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_49 2024-05-17T15:13:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-05-17T15:13:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:13:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:13:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:13:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:13:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_49 2024-05-17T15:13:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:13:11.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:13:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:13:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:13:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_49 128M 2024-05-17T15:13:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:13:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:13:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:13:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:13:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_49 2024-05-17T15:13:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:13:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:13:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:13:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:13:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:13:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_49 2024-05-17T15:13:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:13:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:13:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:13:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:13:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-05-17T15:13:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:13:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:13:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_49 2024-05-17T15:13:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:13:11.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-05-17T15:13:12.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2024-05-17T15:13:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:13:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-05-17T15:13:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_49': 2024-05-17T15:13:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:13:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:13:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:13:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 149bd696b9ba 2024-05-17T15:13:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.149bd696b9ba 2024-05-17T15:13:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:13:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:13:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:13:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:13:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:13:11 2024 2024-05-17T15:13:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:13:11 2024 2024-05-17T15:13:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:13:11 2024 2024-05-17T15:13:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:13:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:13:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 6c86503a-19c3-470e-81c4-44311421ecfd 2024-05-17T15:13:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:13:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:13:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_49 2024-05-17T15:13:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:13:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:13:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_49 2024-05-17T15:13:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:13:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=9 2024-05-17T15:13:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:13:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:13:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 23600128 bytes 2024-05-17T15:13:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 33134592 bytes 2024-05-17T15:13:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 67451904 bytes 2024-05-17T15:13:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 75233280 bytes 2024-05-17T15:13:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 90392576 bytes 2024-05-17T15:13:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 143772672 bytes 2024-05-17T15:13:13.229 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 160962560 bytes 2024-05-17T15:13:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 192993280 bytes 2024-05-17T15:13:13.394 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 232387584 bytes 2024-05-17T15:13:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 267680768 bytes 2024-05-17T15:13:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 319199232 bytes 2024-05-17T15:13:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 384257024 bytes 2024-05-17T15:13:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 416210944 bytes 2024-05-17T15:13:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 468829184 bytes 2024-05-17T15:13:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 533514240 bytes 2024-05-17T15:13:13.990 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4397786357624135) > weight (0.25) 2024-05-17T15:13:13.991 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.2587401597891107) > weight (0.25) 2024-05-17T15:13:13.991 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8698915927093341) > weight (0.25) 2024-05-17T15:13:13.991 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-05-17T15:13:13.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-05-17T15:13:13.991 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-05-17T15:13:13.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi195.stderr:2024-05-17T15:13:13.989+0000 7f482cb34640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 48530) UID: 1000 2024-05-17T15:13:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 563428352 bytes 2024-05-17T15:13:14.136 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 623629312 bytes 2024-05-17T15:13:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 665028608 bytes 2024-05-17T15:13:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 665621504 bytes 2024-05-17T15:13:14.337 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 680908800 bytes 2024-05-17T15:13:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 728019968 bytes 2024-05-17T15:13:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 769253376 bytes 2024-05-17T15:13:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 777810944 bytes 2024-05-17T15:13:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 825492480 bytes 2024-05-17T15:13:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 942134272 bytes 2024-05-17T15:13:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 999394304 bytes 2024-05-17T15:13:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1047845888 bytes 2024-05-17T15:13:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1081686016 bytes 2024-05-17T15:13:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1108620288 bytes 2024-05-17T15:13:15.225 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1124120576 bytes 2024-05-17T15:13:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1147807744 bytes 2024-05-17T15:13:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1169645568 bytes 2024-05-17T15:13:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1247532032 bytes 2024-05-17T15:13:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1319254016 bytes 2024-05-17T15:13:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1324338176 bytes 2024-05-17T15:13:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1329190912 bytes 2024-05-17T15:13:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1401722880 bytes 2024-05-17T15:13:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1517808640 bytes 2024-05-17T15:13:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1536277504 bytes 2024-05-17T15:13:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1581911040 bytes 2024-05-17T15:13:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 1662050304 bytes 2024-05-17T15:13:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 1693161472 bytes 2024-05-17T15:13:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 1751271424 bytes 2024-05-17T15:13:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 1829606400 bytes 2024-05-17T15:13:16.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 1871098880 bytes 2024-05-17T15:13:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 1922148352 bytes 2024-05-17T15:13:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 1970102272 bytes 2024-05-17T15:13:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 1977300992 bytes 2024-05-17T15:13:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2018751488 bytes 2024-05-17T15:13:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2036005888 bytes 2024-05-17T15:13:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2090124288 bytes 2024-05-17T15:13:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2111926272 bytes 2024-05-17T15:13:17.272 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2142030848 bytes 2024-05-17T15:13:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2176012288 bytes 2024-05-17T15:13:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2206034944 bytes 2024-05-17T15:13:17.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2024-05-17T15:13:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2224172032 bytes 2024-05-17T15:13:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2298001408 bytes 2024-05-17T15:13:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2343151616 bytes 2024-05-17T15:13:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2395870208 bytes 2024-05-17T15:13:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2446338048 bytes 2024-05-17T15:13:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2480015360 bytes 2024-05-17T15:13:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2562624512 bytes 2024-05-17T15:13:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2604712960 bytes 2024-05-17T15:13:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2627288064 bytes 2024-05-17T15:13:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2758604800 bytes 2024-05-17T15:13:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2882227200 bytes 2024-05-17T15:13:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6700 IOs, 2882839552 bytes 2024-05-17T15:13:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6800 IOs, 2968526848 bytes 2024-05-17T15:13:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6900 IOs, 3019082752 bytes 2024-05-17T15:13:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7000 IOs, 3041037312 bytes 2024-05-17T15:13:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7100 IOs, 3095370752 bytes 2024-05-17T15:13:19.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7200 IOs, 3163240448 bytes 2024-05-17T15:13:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7300 IOs, 3235441664 bytes 2024-05-17T15:13:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7400 IOs, 3284178944 bytes 2024-05-17T15:13:19.395 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7500 IOs, 3319680000 bytes 2024-05-17T15:13:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7600 IOs, 3373024256 bytes 2024-05-17T15:13:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7700 IOs, 3389829120 bytes 2024-05-17T15:13:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7800 IOs, 3452476416 bytes 2024-05-17T15:13:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:7900 IOs, 3483843584 bytes 2024-05-17T15:13:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8000 IOs, 3560107008 bytes 2024-05-17T15:13:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8100 IOs, 3634035712 bytes 2024-05-17T15:13:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8200 IOs, 3646844928 bytes 2024-05-17T15:13:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8300 IOs, 3649987584 bytes 2024-05-17T15:13:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8400 IOs, 3684002816 bytes 2024-05-17T15:13:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8500 IOs, 3749234688 bytes 2024-05-17T15:13:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8600 IOs, 3770784768 bytes 2024-05-17T15:13:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8700 IOs, 3806113792 bytes 2024-05-17T15:13:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8800 IOs, 3846763520 bytes 2024-05-17T15:13:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:8900 IOs, 3932337152 bytes 2024-05-17T15:13:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9000 IOs, 3943261184 bytes 2024-05-17T15:13:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9100 IOs, 3944305664 bytes 2024-05-17T15:13:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9200 IOs, 3976471552 bytes 2024-05-17T15:13:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9300 IOs, 3997694976 bytes 2024-05-17T15:13:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9400 IOs, 4030855168 bytes 2024-05-17T15:13:20.992 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-05-17T15:13:20.992 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:13:20.992 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-05-17T15:13:20.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-05-17T15:13:20.993 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-05-17T15:13:20.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-05-17T15:13:20.996 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-05-17T15:13:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9500 IOs, 4058169344 bytes 2024-05-17T15:13:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9600 IOs, 4067896320 bytes 2024-05-17T15:13:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9700 IOs, 4106786816 bytes 2024-05-17T15:13:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9800 IOs, 4172626944 bytes 2024-05-17T15:13:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:9900 IOs, 4184198144 bytes 2024-05-17T15:13:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10000 IOs, 4216088576 bytes 2024-05-17T15:13:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:10100 IOs, 4303556608 bytes 2024-05-17T15:13:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:13:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:13:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:13:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:13:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:13:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_50 2024-05-17T15:13:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-05-17T15:13:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:13:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:13:21.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:13:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:13:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_50 2024-05-17T15:13:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:13:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:13:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n snapshot ']' 2024-05-17T15:13:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:13:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_image cluster2 mirror image_50 128M 2024-05-17T15:13:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:13:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:13:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:13:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:13:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_50 2024-05-17T15:13:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:13:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local size=128 2024-05-17T15:13:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n 128M ']' 2024-05-17T15:13:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ size=128M 2024-05-17T15:13:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ shift 2024-05-17T15:13:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_50 2024-05-17T15:13:21.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ is_pool_mirror_mode_image mirror 2024-05-17T15:13:21.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:13:21.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' image = image ']' 2024-05-17T15:13:21.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:13:21.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-05-17T15:13:21.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:13:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:13:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_50 2024-05-17T15:13:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local mode=snapshot 2024-05-17T15:13:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-05-17T15:13:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirroring enabled 2024-05-17T15:13:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-05-17T15:13:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_50': 2024-05-17T15:13:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:13:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:13:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:13:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 14b09e29da4f 2024-05-17T15:13:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.14b09e29da4f 2024-05-17T15:13:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:13:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-05-17T15:13:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:13:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:13:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:13:21 2024 2024-05-17T15:13:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:13:21 2024 2024-05-17T15:13:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:13:21 2024 2024-05-17T15:13:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:13:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:13:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 9ca9b16d-6fb0-46d4-9998-208fa4980f4f 2024-05-17T15:13:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: true 2024-05-17T15:13:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -n '' ']' 2024-05-17T15:13:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stress_write_image cluster2 mirror image_50 2024-05-17T15:13:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:13:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:13:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_50 2024-05-17T15:13:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-05-17T15:13:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local duration=7 2024-05-17T15:13:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:13:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-05-17T15:13:22.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2024-05-17T15:13:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:100 IOs, 82545664 bytes 2024-05-17T15:13:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:200 IOs, 83167232 bytes 2024-05-17T15:13:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:300 IOs, 154625024 bytes 2024-05-17T15:13:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:400 IOs, 195416064 bytes 2024-05-17T15:13:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:500 IOs, 219338752 bytes 2024-05-17T15:13:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:600 IOs, 241600512 bytes 2024-05-17T15:13:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:700 IOs, 285357056 bytes 2024-05-17T15:13:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:800 IOs, 333291520 bytes 2024-05-17T15:13:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:900 IOs, 441404416 bytes 2024-05-17T15:13:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1000 IOs, 475009024 bytes 2024-05-17T15:13:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1100 IOs, 512441344 bytes 2024-05-17T15:13:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1200 IOs, 516939776 bytes 2024-05-17T15:13:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1300 IOs, 593105920 bytes 2024-05-17T15:13:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1400 IOs, 667327488 bytes 2024-05-17T15:13:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1500 IOs, 720223232 bytes 2024-05-17T15:13:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1600 IOs, 739847168 bytes 2024-05-17T15:13:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1700 IOs, 790559744 bytes 2024-05-17T15:13:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1800 IOs, 836194304 bytes 2024-05-17T15:13:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:1900 IOs, 899143680 bytes 2024-05-17T15:13:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2000 IOs, 994638848 bytes 2024-05-17T15:13:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2100 IOs, 1053771776 bytes 2024-05-17T15:13:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2200 IOs, 1113773056 bytes 2024-05-17T15:13:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2300 IOs, 1145901056 bytes 2024-05-17T15:13:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2400 IOs, 1178330112 bytes 2024-05-17T15:13:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2500 IOs, 1220885504 bytes 2024-05-17T15:13:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2600 IOs, 1274030080 bytes 2024-05-17T15:13:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2700 IOs, 1374406656 bytes 2024-05-17T15:13:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2800 IOs, 1416222720 bytes 2024-05-17T15:13:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:2900 IOs, 1485505536 bytes 2024-05-17T15:13:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3000 IOs, 1599024128 bytes 2024-05-17T15:13:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3100 IOs, 1653947392 bytes 2024-05-17T15:13:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3200 IOs, 1660988416 bytes 2024-05-17T15:13:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3300 IOs, 1683201024 bytes 2024-05-17T15:13:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3400 IOs, 1684787200 bytes 2024-05-17T15:13:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3500 IOs, 1745150976 bytes 2024-05-17T15:13:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3600 IOs, 1802621952 bytes 2024-05-17T15:13:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3700 IOs, 1841009664 bytes 2024-05-17T15:13:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3800 IOs, 1864543232 bytes 2024-05-17T15:13:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:3900 IOs, 1906136064 bytes 2024-05-17T15:13:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4000 IOs, 1941996544 bytes 2024-05-17T15:13:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4100 IOs, 2027060224 bytes 2024-05-17T15:13:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4200 IOs, 2059296768 bytes 2024-05-17T15:13:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4300 IOs, 2089544704 bytes 2024-05-17T15:13:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4400 IOs, 2128058368 bytes 2024-05-17T15:13:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4500 IOs, 2165055488 bytes 2024-05-17T15:13:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4600 IOs, 2188181504 bytes 2024-05-17T15:13:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4700 IOs, 2208497664 bytes 2024-05-17T15:13:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4800 IOs, 2266297344 bytes 2024-05-17T15:13:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:4900 IOs, 2302178304 bytes 2024-05-17T15:13:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5000 IOs, 2351632384 bytes 2024-05-17T15:13:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5100 IOs, 2439709696 bytes 2024-05-17T15:13:28.277 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2024-05-17T15:13:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5200 IOs, 2488041472 bytes 2024-05-17T15:13:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5300 IOs, 2522294272 bytes 2024-05-17T15:13:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5400 IOs, 2591389696 bytes 2024-05-17T15:13:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5500 IOs, 2645395456 bytes 2024-05-17T15:13:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5600 IOs, 2646024192 bytes 2024-05-17T15:13:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5700 IOs, 2648428544 bytes 2024-05-17T15:13:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5800 IOs, 2686602240 bytes 2024-05-17T15:13:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:5900 IOs, 2687224832 bytes 2024-05-17T15:13:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6000 IOs, 2719845376 bytes 2024-05-17T15:13:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6100 IOs, 2775411712 bytes 2024-05-17T15:13:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6200 IOs, 2822489088 bytes 2024-05-17T15:13:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6300 IOs, 2896487424 bytes 2024-05-17T15:13:29.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6400 IOs, 2920859648 bytes 2024-05-17T15:13:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6500 IOs, 2962640896 bytes 2024-05-17T15:13:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:6600 IOs, 2997026816 bytes 2024-05-17T15:13:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ error_code=124 2024-05-17T15:13:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set -e 2024-05-17T15:13:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 124 -eq 124 ']' 2024-05-17T15:13:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:13:29.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-05-17T15:13:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:13:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:13:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image_count=50 2024-05-17T15:13:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:13:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local count 2024-05-17T15:13:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_count=0 2024-05-17T15:13:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:13:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ seq 1 40 2024-05-17T15:13:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in `seq 1 40` 2024-05-17T15:13:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 1 -ne 1 2024-05-17T15:13:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-05-17T15:13:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep 'images: ' 2024-05-17T15:13:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cut '-d ' -f 2 2024-05-17T15:13:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: failed to query services: (13) Permission denied 2024-05-17T15:13:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ count=50 2024-05-17T15:13:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 50 = 50 2024-05-17T15:13:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:13:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-05-17T15:13:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:13:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:13:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:13:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state 2024-05-17T15:13:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ seq 1 40 2024-05-17T15:13:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in `seq 1 40` 2024-05-17T15:13:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 1 -ne 1 2024-05-17T15:13:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-05-17T15:13:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep 'image health:' 2024-05-17T15:13:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cut '-d ' -f 3 2024-05-17T15:13:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: failed to query services: (13) Permission denied 2024-05-17T15:13:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ state=OK 2024-05-17T15:13:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test OK = ERROR 2024-05-17T15:13:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test OK = OK 2024-05-17T15:13:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:13:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-05-17T15:13:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:13:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:13:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image_count=50 2024-05-17T15:13:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:13:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local count 2024-05-17T15:13:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local last_count=0 2024-05-17T15:13:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:13:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ seq 1 40 2024-05-17T15:13:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in `seq 1 40` 2024-05-17T15:13:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 1 -ne 1 2024-05-17T15:13:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-05-17T15:13:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep 'images: ' 2024-05-17T15:13:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cut '-d ' -f 2 2024-05-17T15:13:29.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: failed to query services: (13) Permission denied 2024-05-17T15:13:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ count=50 2024-05-17T15:13:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 50 = 50 2024-05-17T15:13:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:13:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-05-17T15:13:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:13:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:13:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:13:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state 2024-05-17T15:13:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ seq 1 40 2024-05-17T15:13:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in `seq 1 40` 2024-05-17T15:13:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 1 -ne 1 2024-05-17T15:13:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-05-17T15:13:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep 'image health:' 2024-05-17T15:13:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cut '-d ' -f 3 2024-05-17T15:13:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: failed to query services: (13) Permission denied 2024-05-17T15:13:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ state=WARNING 2024-05-17T15:13:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test WARNING = ERROR 2024-05-17T15:13:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test WARNING = OK 2024-05-17T15:13:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in `seq 1 40` 2024-05-17T15:13:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 2 -ne 1 2024-05-17T15:13:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 30 2024-05-17T15:13:33.681 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2024-05-17T15:13:34.882 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-05-17T15:13:34.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-05-17T15:13:34.886 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-05-17T15:13:34.886 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-05-17T15:13:34.886 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:13:34.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T15:13:34.881+0000 7fcda7878640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 49956) UID: 1000 2024-05-17T15:13:39.084 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2024-05-17T15:13:40.988 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-05-17T15:13:40.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-05-17T15:13:40.989 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-05-17T15:13:40.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-05-17T15:13:40.992 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2024-05-17T15:13:44.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2024-05-17T15:13:49.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2024-05-17T15:13:55.299 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2024-05-17T15:13:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-05-17T15:13:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep 'image health:' 2024-05-17T15:13:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cut '-d ' -f 3 2024-05-17T15:13:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: failed to query services: (13) Permission denied 2024-05-17T15:13:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ state=OK 2024-05-17T15:13:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test OK = ERROR 2024-05-17T15:13:59.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test OK = OK 2024-05-17T15:13:59.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:13:59.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ testlog 'TEST: compare many images' 2024-05-17T15:13:59.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ tee -a /tmp/tmp.0i1dljubcH/rbd-mirror.test.log 2024-05-17T15:13:59.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ date '+%F %T' 2024-05-17T15:13:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17 15:13:59 TEST: compare many images 2024-05-17T15:13:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:2024-05-17 15:13:59 TEST: compare many images 2024-05-17T15:13:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ seq 1 50 2024-05-17T15:13:59.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:13:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_1 2024-05-17T15:13:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_1 snap 2024-05-17T15:13:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:13:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:13:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:13:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:13:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:14:00.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2024-05-17T15:14:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-05-17T15:14:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:14:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-05-17T15:14:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:14:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:14:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:14:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:14:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:14:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-05-17T15:14:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:14:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:14:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:14:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:14:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-05-17T15:14:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:14:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:14:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:14:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:14:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:14:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:14:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:14:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:14:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:14:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:14:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-05-17T15:14:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:14:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:14:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:14:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:14:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:14:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:14:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:14:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:14:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:14:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:14:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:14:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:14:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:14:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:14:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:14:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:14:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:02.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:02.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-05-17T15:14:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:no valid command found; 2 closest matches: 2024-05-17T15:14:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status mirror/image_13 2024-05-17T15:14:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status mirror/image_19 2024-05-17T15:14:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:14:02.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:14:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-05-17T15:14:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:14:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:14:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:14:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:14:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:14:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:14:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:14:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:14:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:14:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:14:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:14:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:14:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:14:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:14:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:14:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:14:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:14:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:14:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:14:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:14:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:14:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:14:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:14:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:14:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:14:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:14:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:14:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:14:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:14:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:14:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-05-17T15:14:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:no valid command found; 3 closest matches: 2024-05-17T15:14:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status mirror/image_11 2024-05-17T15:14:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status mirror/image_14 2024-05-17T15:14:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status mirror/image_16 2024-05-17T15:14:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:14:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:14:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_1 2024-05-17T15:14:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:14:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:14:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:14:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:14:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:14:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:14:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:14:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:14:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:14:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:14:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:14:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:14:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:14:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:14:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:14:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:14:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:14:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:14:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:14:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:14:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:14:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:14:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:14:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:14:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:14:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:14:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:14:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:14:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:14:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:14:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_1 2024-05-17T15:14:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:14:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:14:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:14:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:14:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:14:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:14:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:14:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-05-17T15:14:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:14:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:14:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:14:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:14:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2024-05-17T15:14:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:14:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:14:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_1.status 2024-05-17T15:14:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_1.status 2024-05-17T15:14:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:14:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_1.status 2024-05-17T15:14:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_1.status 2024-05-17T15:14:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_1.status 2024-05-17T15:14:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:14:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_1.status 2024-05-17T15:14:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-05-17T15:14:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:14:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-05-17T15:14:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 86 2024-05-17T15:14:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_1.status 2024-05-17T15:14:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:14:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_1.status 2024-05-17T15:14:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-05-17T15:14:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:03.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:14:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=86 2024-05-17T15:14:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:14:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_1.status 2024-05-17T15:14:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:14:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_1.status 2024-05-17T15:14:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-05-17T15:14:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:04.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=35 2024-05-17T15:14:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 86 = 35 2024-05-17T15:14:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:14:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_1.status 2024-05-17T15:14:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:14:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_1.status 2024-05-17T15:14:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-05-17T15:14:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:14:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:14:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:14:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:14:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 86 = '' 2024-05-17T15:14:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:14:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_1.status 2024-05-17T15:14:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:14:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_1.status 2024-05-17T15:14:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-05-17T15:14:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:14:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:14:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:14:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:14:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 86 = '' 2024-05-17T15:14:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:14:06.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2024-05-17T15:14:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_1.status 2024-05-17T15:14:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:14:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_1.status 2024-05-17T15:14:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-05-17T15:14:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=86 2024-05-17T15:14:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 86 = 86 2024-05-17T15:14:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-05-17T15:14:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:14:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:14:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:14:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:14:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-05-17T15:14:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_1': 2024-05-17T15:14:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:14:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:14:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:14:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10c8f0e8d377 2024-05-17T15:14:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10c8f0e8d377 2024-05-17T15:14:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:14:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:14:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:14:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:14:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:04:43 2024 2024-05-17T15:14:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:04:43 2024 2024-05-17T15:14:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:04:43 2024 2024-05-17T15:14:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:14:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:14:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:14:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: ceaa7752-4708-4453-aad7-8c9fe4b5ad62 2024-05-17T15:14:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:14:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_1 snap 2024-05-17T15:14:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_1 2024-05-17T15:14:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:14:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_1.export 2024-05-17T15:14:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_1.export 2024-05-17T15:14:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_1.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_1.export 2024-05-17T15:14:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_1.export 2024-05-17T15:14:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:14:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_1.export 2024-05-17T15:14:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:14:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_1.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_1.export 2024-05-17T15:14:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_1.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_1.export 2024-05-17T15:14:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:14:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_2 2024-05-17T15:14:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_2 snap 2024-05-17T15:14:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:14:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:14:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-05-17T15:14:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:14:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-05-17T15:14:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:14:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:14:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:14:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:14:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:14:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-05-17T15:14:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:14:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:14:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:14:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:14:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-05-17T15:14:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:14:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:14:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:14:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:14:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:14:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:14:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:14:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:14:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:14:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:14:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-05-17T15:14:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:14:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:14:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:14:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:14:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:14:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:14:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:14:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:14:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:14:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:14:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:14:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:14:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:14:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:14:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:14:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:14:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:14:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:14:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:14:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:14:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:14:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:14:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:14:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:14:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:14:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:14:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:14:10.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:10.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:10.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:10.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-05-17T15:14:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:14:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:14:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:14:11.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:14:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:14:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:14:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:14:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-05-17T15:14:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:14:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:14:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:14:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:14:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2024-05-17T15:14:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:14:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:14:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_2.status 2024-05-17T15:14:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_2.status 2024-05-17T15:14:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:14:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_2.status 2024-05-17T15:14:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_2.status 2024-05-17T15:14:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_2.status 2024-05-17T15:14:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:14:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_2.status 2024-05-17T15:14:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-05-17T15:14:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:14:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-05-17T15:14:11.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2024-05-17T15:14:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 88 2024-05-17T15:14:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_2.status 2024-05-17T15:14:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:14:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_2.status 2024-05-17T15:14:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-05-17T15:14:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:14:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=88 2024-05-17T15:14:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:14:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_2.status 2024-05-17T15:14:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:14:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_2.status 2024-05-17T15:14:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-05-17T15:14:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=36 2024-05-17T15:14:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 88 = 36 2024-05-17T15:14:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:14:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_2.status 2024-05-17T15:14:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:14:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_2.status 2024-05-17T15:14:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-05-17T15:14:12.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:14:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:14:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:14:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:14:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 88 = '' 2024-05-17T15:14:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:14:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_2.status 2024-05-17T15:14:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:14:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_2.status 2024-05-17T15:14:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-05-17T15:14:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:14:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:14:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:14:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:14:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 88 = '' 2024-05-17T15:14:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:14:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_2.status 2024-05-17T15:14:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:14:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_2.status 2024-05-17T15:14:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-05-17T15:14:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=88 2024-05-17T15:14:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 88 = 88 2024-05-17T15:14:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-05-17T15:14:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:14:15.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:15.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:14:15.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:14:15.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:14:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-05-17T15:14:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_2': 2024-05-17T15:14:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:14:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:14:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:14:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d3b6d76de2 2024-05-17T15:14:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d3b6d76de2 2024-05-17T15:14:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:14:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:14:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:14:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:14:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:04:54 2024 2024-05-17T15:14:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:04:54 2024 2024-05-17T15:14:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:04:54 2024 2024-05-17T15:14:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:14:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:14:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:14:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 8d44a27c-5bdd-4a78-9b8a-9a4d22d34367 2024-05-17T15:14:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:14:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_2 snap 2024-05-17T15:14:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_2 2024-05-17T15:14:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:14:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_2.export 2024-05-17T15:14:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_2.export 2024-05-17T15:14:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_2.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_2.export 2024-05-17T15:14:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_2.export 2024-05-17T15:14:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:14:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_2.export 2024-05-17T15:14:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:14:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_2.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_2.export 2024-05-17T15:14:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_2.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_2.export 2024-05-17T15:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_3 2024-05-17T15:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_3 snap 2024-05-17T15:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:14:16.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2024-05-17T15:14:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-05-17T15:14:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:14:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-05-17T15:14:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:14:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:14:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:14:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:14:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:14:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-05-17T15:14:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:14:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:14:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:14:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:14:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-05-17T15:14:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:14:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:14:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:14:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:14:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:14:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:14:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:14:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:14:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:14:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:14:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-05-17T15:14:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:14:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:14:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:14:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:14:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:14:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:14:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:14:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:14:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:14:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:14:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:14:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:14:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:14:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:14:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:14:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:14:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:14:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:18.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-05-17T15:14:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:no valid command found; 3 closest matches: 2024-05-17T15:14:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status mirror/image_30 2024-05-17T15:14:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status mirror/image_31 2024-05-17T15:14:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status mirror/image_32 2024-05-17T15:14:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:14:19.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:14:19.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2024-05-17T15:14:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:14:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:14:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:14:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:14:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:14:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:14:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:14:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:14:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:14:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:14:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:14:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:14:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:14:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:14:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:14:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:14:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:14:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:14:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:14:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:14:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:14:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:14:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:14:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:14:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:14:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:14:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:14:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:14:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:14:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:14:19.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2024-05-17T15:14:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:no valid command found; 2 closest matches: 2024-05-17T15:14:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status mirror/image_35 2024-05-17T15:14:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status mirror/image_39 2024-05-17T15:14:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:14:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:14:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2024-05-17T15:14:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:14:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:14:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:14:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:14:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:14:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:14:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:14:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:14:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:14:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:14:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:14:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:14:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:14:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:14:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:14:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:14:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:14:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:14:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:14:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:14:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:14:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:14:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:14:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:14:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:14:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:14:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:14:19.144 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:14:19.144 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:14:19.144 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:14:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2024-05-17T15:14:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:14:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:14:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:14:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:14:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:14:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:14:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:14:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-05-17T15:14:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:14:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:14:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:14:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:14:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2024-05-17T15:14:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:14:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:14:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_3.status 2024-05-17T15:14:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_3.status 2024-05-17T15:14:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:14:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_3.status 2024-05-17T15:14:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_3.status 2024-05-17T15:14:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_3.status 2024-05-17T15:14:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:14:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_3.status 2024-05-17T15:14:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-05-17T15:14:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:14:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-05-17T15:14:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 90 2024-05-17T15:14:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_3.status 2024-05-17T15:14:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:14:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_3.status 2024-05-17T15:14:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-05-17T15:14:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:14:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=90 2024-05-17T15:14:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:14:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_3.status 2024-05-17T15:14:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:14:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_3.status 2024-05-17T15:14:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-05-17T15:14:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=37 2024-05-17T15:14:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 90 = 37 2024-05-17T15:14:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:14:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_3.status 2024-05-17T15:14:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:14:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_3.status 2024-05-17T15:14:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-05-17T15:14:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:14:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:14:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:14:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:14:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 90 = '' 2024-05-17T15:14:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:14:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_3.status 2024-05-17T15:14:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:14:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_3.status 2024-05-17T15:14:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-05-17T15:14:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:14:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:14:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:14:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:14:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 90 = '' 2024-05-17T15:14:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:21.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:14:22.319 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2024-05-17T15:14:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_3.status 2024-05-17T15:14:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:14:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_3.status 2024-05-17T15:14:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-05-17T15:14:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=90 2024-05-17T15:14:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 90 = 90 2024-05-17T15:14:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-05-17T15:14:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:23.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:14:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:14:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:14:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:14:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-05-17T15:14:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_3': 2024-05-17T15:14:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:14:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:14:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:14:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 11358d942a71 2024-05-17T15:14:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.11358d942a71 2024-05-17T15:14:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:14:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:14:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:14:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:14:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:02 2024 2024-05-17T15:14:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:05:02 2024 2024-05-17T15:14:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:02 2024 2024-05-17T15:14:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:14:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:14:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:14:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 5191746b-1578-4957-a477-41bde526d1e0 2024-05-17T15:14:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:14:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_3 snap 2024-05-17T15:14:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_3 2024-05-17T15:14:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:14:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_3.export 2024-05-17T15:14:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_3.export 2024-05-17T15:14:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_3.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_3.export 2024-05-17T15:14:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_3.export 2024-05-17T15:14:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:14:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_3.export 2024-05-17T15:14:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:14:24.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_3.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_3.export 2024-05-17T15:14:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_3.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_3.export 2024-05-17T15:14:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:14:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_4 2024-05-17T15:14:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_4 snap 2024-05-17T15:14:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:14:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:14:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-05-17T15:14:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:14:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-05-17T15:14:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:14:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:14:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:14:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:14:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:14:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-05-17T15:14:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:14:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:14:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:14:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:14:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-05-17T15:14:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:14:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:14:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:14:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:14:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:14:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:14:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:14:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:14:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:14:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:14:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-05-17T15:14:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:14:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:14:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:14:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:14:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:14:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:14:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:14:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:14:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:14:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:14:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:14:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:14:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:14:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:14:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-05-17T15:14:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:no valid command found; 1 closest matches: 2024-05-17T15:14:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status mirror/image_44 2024-05-17T15:14:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:14:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:14:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-05-17T15:14:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:14:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:14:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:14:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:14:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:14:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:14:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:14:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:14:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:14:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:14:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:14:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:14:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:14:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:14:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:14:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:14:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:14:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:14:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:14:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:14:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:14:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:14:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:14:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:14:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:14:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:14:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:14:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:14:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:14:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:14:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-05-17T15:14:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:14:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:14:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:14:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:14:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:14:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:14:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:14:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-05-17T15:14:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:14:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:14:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:14:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:14:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2024-05-17T15:14:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:14:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:14:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_4.status 2024-05-17T15:14:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_4.status 2024-05-17T15:14:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:14:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_4.status 2024-05-17T15:14:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_4.status 2024-05-17T15:14:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_4.status 2024-05-17T15:14:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:14:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_4.status 2024-05-17T15:14:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-05-17T15:14:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:14:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-05-17T15:14:27.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2024-05-17T15:14:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 92 2024-05-17T15:14:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_4.status 2024-05-17T15:14:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:14:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_4.status 2024-05-17T15:14:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-05-17T15:14:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:14:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=92 2024-05-17T15:14:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:14:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_4.status 2024-05-17T15:14:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:14:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_4.status 2024-05-17T15:14:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-05-17T15:14:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=38 2024-05-17T15:14:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 92 = 38 2024-05-17T15:14:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:14:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_4.status 2024-05-17T15:14:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:14:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_4.status 2024-05-17T15:14:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-05-17T15:14:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:14:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:14:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:14:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:14:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 92 = '' 2024-05-17T15:14:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:14:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_4.status 2024-05-17T15:14:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:14:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_4.status 2024-05-17T15:14:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-05-17T15:14:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:14:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:14:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:14:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:14:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 92 = '' 2024-05-17T15:14:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:14:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_4.status 2024-05-17T15:14:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:14:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_4.status 2024-05-17T15:14:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-05-17T15:14:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=92 2024-05-17T15:14:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 92 = 92 2024-05-17T15:14:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-05-17T15:14:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:14:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:14:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:14:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:14:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-05-17T15:14:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_4': 2024-05-17T15:14:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:14:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:14:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:14:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 121df7b93334 2024-05-17T15:14:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.121df7b93334 2024-05-17T15:14:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:14:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:14:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:14:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:14:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:10 2024 2024-05-17T15:14:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:05:10 2024 2024-05-17T15:14:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:10 2024 2024-05-17T15:14:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:14:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:14:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:14:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 1fda3480-fa61-4b4b-93a6-2dd063ac50f4 2024-05-17T15:14:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:14:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_4 snap 2024-05-17T15:14:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_4 2024-05-17T15:14:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:14:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_4.export 2024-05-17T15:14:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_4.export 2024-05-17T15:14:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_4.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_4.export 2024-05-17T15:14:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_4.export 2024-05-17T15:14:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:14:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_4.export 2024-05-17T15:14:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:14:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_4.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_4.export 2024-05-17T15:14:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_4.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_4.export 2024-05-17T15:14:32.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:32.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:14:32.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_5 2024-05-17T15:14:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_5 snap 2024-05-17T15:14:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:14:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:14:33.127 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2024-05-17T15:14:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-05-17T15:14:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:14:34.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-05-17T15:14:34.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:34.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:34.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:14:34.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:14:34.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:14:34.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:14:34.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:14:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-05-17T15:14:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:14:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:14:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:14:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:14:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-05-17T15:14:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:14:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:14:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:14:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:14:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:14:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:14:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:14:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:14:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:14:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:14:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-05-17T15:14:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:14:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:14:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:14:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:14:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:14:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:14:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:14:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:14:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:14:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:14:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:14:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:14:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:14:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:14:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:14:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:14:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:14:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:14:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:14:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:14:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:14:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:14:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:14:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:14:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:14:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:14:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:14:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-05-17T15:14:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:14:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:14:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:14:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:14:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:14:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:14:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:14:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-05-17T15:14:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:14:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:14:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:14:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:14:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2024-05-17T15:14:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:14:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:14:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_5.status 2024-05-17T15:14:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_5.status 2024-05-17T15:14:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:14:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_5.status 2024-05-17T15:14:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_5.status 2024-05-17T15:14:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_5.status 2024-05-17T15:14:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:14:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_5.status 2024-05-17T15:14:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-05-17T15:14:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:14:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-05-17T15:14:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 94 2024-05-17T15:14:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_5.status 2024-05-17T15:14:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:14:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_5.status 2024-05-17T15:14:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-05-17T15:14:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:14:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=94 2024-05-17T15:14:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:14:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_5.status 2024-05-17T15:14:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:14:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_5.status 2024-05-17T15:14:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-05-17T15:14:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=39 2024-05-17T15:14:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 94 = 39 2024-05-17T15:14:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:14:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_5.status 2024-05-17T15:14:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:14:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_5.status 2024-05-17T15:14:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-05-17T15:14:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:37.342 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:14:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:14:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:14:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:14:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 94 = '' 2024-05-17T15:14:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:14:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_5.status 2024-05-17T15:14:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:14:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_5.status 2024-05-17T15:14:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-05-17T15:14:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:14:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:14:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:14:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:14:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 94 = '' 2024-05-17T15:14:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:14:38.391 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:14:38.389+0000 7f7a4dbdb640 -1 reset not still connected to 0x55d90a0b7380 2024-05-17T15:14:38.391 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:14:38.389+0000 7f7a4dbdb640 -1 reset not still connected to 0x55d914920f70 2024-05-17T15:14:38.391 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:14:38.389+0000 7f7a4dbdb640 -1 reset not still connected to 0x55d9245b4d00 2024-05-17T15:14:38.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~86s 2024-05-17T15:14:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_5.status 2024-05-17T15:14:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:14:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_5.status 2024-05-17T15:14:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-05-17T15:14:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=94 2024-05-17T15:14:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 94 = 94 2024-05-17T15:14:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-05-17T15:14:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:14:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:14:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:14:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:14:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-05-17T15:14:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_5': 2024-05-17T15:14:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:14:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:14:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:14:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10c8713e805a 2024-05-17T15:14:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10c8713e805a 2024-05-17T15:14:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:14:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:14:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:14:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:14:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:17 2024 2024-05-17T15:14:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:05:17 2024 2024-05-17T15:14:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:17 2024 2024-05-17T15:14:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:14:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:14:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:14:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 82562a71-d364-4946-b5e9-67440f1071c4 2024-05-17T15:14:40.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:14:40.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_5 snap 2024-05-17T15:14:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_5 2024-05-17T15:14:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:14:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_5.export 2024-05-17T15:14:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_5.export 2024-05-17T15:14:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_5.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_5.export 2024-05-17T15:14:40.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_5.export 2024-05-17T15:14:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:14:41.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_5.export 2024-05-17T15:14:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:14:41.349 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_5.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_5.export 2024-05-17T15:14:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_5.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_5.export 2024-05-17T15:14:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:14:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_6 2024-05-17T15:14:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_6 snap 2024-05-17T15:14:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:41.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:41.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:14:41.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:41.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:14:42.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-05-17T15:14:42.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:42.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:42.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:14:42.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-05-17T15:14:42.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:14:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:14:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:14:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:14:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:14:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-05-17T15:14:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:14:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:14:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:14:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:14:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-05-17T15:14:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:14:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:14:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:14:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:14:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:14:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:14:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:14:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:14:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:14:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:14:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-05-17T15:14:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:14:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:14:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:14:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:14:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:14:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:14:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:14:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:14:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:14:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:14:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:14:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:14:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:14:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:14:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:14:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:14:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:14:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:14:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:14:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:14:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:14:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:14:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:14:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:14:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:14:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:14:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:14:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-05-17T15:14:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-05-17T15:14:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:14:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:14:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:14:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-05-17T15:14:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:14:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:14:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:14:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:14:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:14:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:14:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:14:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:14:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:14:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:14:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:14:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:14:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:14:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:14:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:14:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:14:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:14:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:14:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:14:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:14:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:14:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:14:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:14:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:14:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:14:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:14:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:14:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:14:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:14:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:14:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-05-17T15:14:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-05-17T15:14:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:14:43.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:14:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:14:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_6 2024-05-17T15:14:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:14:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:14:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:14:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:14:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:14:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:14:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:14:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:14:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:14:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:14:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:14:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:14:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:14:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:14:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:14:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:14:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:14:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:14:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:14:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:14:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:14:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:14:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:14:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:14:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:14:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:14:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:14:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:14:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:14:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:14:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_6 2024-05-17T15:14:43.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2024-05-17T15:14:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:14:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:14:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:14:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:14:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:14:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:14:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:14:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-05-17T15:14:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:14:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:14:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:14:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:14:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2024-05-17T15:14:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:14:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:14:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_6.status 2024-05-17T15:14:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_6.status 2024-05-17T15:14:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:14:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_6.status 2024-05-17T15:14:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_6.status 2024-05-17T15:14:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_6.status 2024-05-17T15:14:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:14:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_6.status 2024-05-17T15:14:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-05-17T15:14:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:14:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-05-17T15:14:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 96 2024-05-17T15:14:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_6.status 2024-05-17T15:14:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:14:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_6.status 2024-05-17T15:14:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-05-17T15:14:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:14:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=96 2024-05-17T15:14:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:14:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_6.status 2024-05-17T15:14:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:14:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_6.status 2024-05-17T15:14:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-05-17T15:14:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=40 2024-05-17T15:14:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 96 = 40 2024-05-17T15:14:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:14:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_6.status 2024-05-17T15:14:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:14:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_6.status 2024-05-17T15:14:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-05-17T15:14:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:45.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:14:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:14:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:14:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:14:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 96 = '' 2024-05-17T15:14:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:14:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_6.status 2024-05-17T15:14:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:14:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_6.status 2024-05-17T15:14:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-05-17T15:14:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:14:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:14:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:14:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:14:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 96 = '' 2024-05-17T15:14:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:14:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_6.status 2024-05-17T15:14:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:14:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_6.status 2024-05-17T15:14:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-05-17T15:14:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=96 2024-05-17T15:14:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 96 = 96 2024-05-17T15:14:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-05-17T15:14:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:14:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:14:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:14:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:14:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-05-17T15:14:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_6': 2024-05-17T15:14:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:14:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:14:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:14:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12844609881d 2024-05-17T15:14:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12844609881d 2024-05-17T15:14:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:14:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:14:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:14:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:14:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:27 2024 2024-05-17T15:14:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:05:27 2024 2024-05-17T15:14:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:27 2024 2024-05-17T15:14:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:14:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:14:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:14:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: b40480e1-3aef-46bb-a86f-05a1ee0d2d26 2024-05-17T15:14:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:14:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_6 snap 2024-05-17T15:14:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_6 2024-05-17T15:14:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:14:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_6.export 2024-05-17T15:14:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_6.export 2024-05-17T15:14:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_6.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_6.export 2024-05-17T15:14:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_6.export 2024-05-17T15:14:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:14:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_6.export 2024-05-17T15:14:49.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2024-05-17T15:14:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:14:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_6.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_6.export 2024-05-17T15:14:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_6.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_6.export 2024-05-17T15:14:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:14:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_7 2024-05-17T15:14:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_7 snap 2024-05-17T15:14:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:14:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:14:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-05-17T15:14:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:14:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-05-17T15:14:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:14:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:14:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:14:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:14:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:14:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-05-17T15:14:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:14:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:14:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:14:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:14:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-05-17T15:14:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:14:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:14:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:14:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:14:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:14:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:14:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:14:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:14:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:14:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:14:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-05-17T15:14:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:14:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:14:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:14:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:14:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:14:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:14:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:14:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:14:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:14:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:14:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:14:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:14:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:14:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:14:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:14:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:14:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:14:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:14:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:14:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:14:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:14:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:14:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:14:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:14:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:14:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:14:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:14:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-05-17T15:14:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:14:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:14:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:14:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:14:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:14:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:14:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:14:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-05-17T15:14:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:14:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:14:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:14:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:14:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2024-05-17T15:14:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:14:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:14:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_7.status 2024-05-17T15:14:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_7.status 2024-05-17T15:14:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:14:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_7.status 2024-05-17T15:14:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_7.status 2024-05-17T15:14:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_7.status 2024-05-17T15:14:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:14:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_7.status 2024-05-17T15:14:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-05-17T15:14:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:14:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-05-17T15:14:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 98 2024-05-17T15:14:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_7.status 2024-05-17T15:14:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:14:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_7.status 2024-05-17T15:14:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-05-17T15:14:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:14:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=98 2024-05-17T15:14:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:14:52.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_7.status 2024-05-17T15:14:52.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:52.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:52.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:14:52.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_7.status 2024-05-17T15:14:52.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-05-17T15:14:52.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=41 2024-05-17T15:14:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 98 = 41 2024-05-17T15:14:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:14:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_7.status 2024-05-17T15:14:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:14:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_7.status 2024-05-17T15:14:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-05-17T15:14:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:14:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:14:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:14:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:14:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 98 = '' 2024-05-17T15:14:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:14:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_7.status 2024-05-17T15:14:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:14:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_7.status 2024-05-17T15:14:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-05-17T15:14:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:14:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:14:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:14:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:14:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:14:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 98 = '' 2024-05-17T15:14:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:14:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:14:54.742 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2024-05-17T15:14:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_7.status 2024-05-17T15:14:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:14:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_7.status 2024-05-17T15:14:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-05-17T15:14:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:14:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:14:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=98 2024-05-17T15:14:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 98 = 98 2024-05-17T15:14:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-05-17T15:14:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:14:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:14:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:14:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:14:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-05-17T15:14:56.996 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-05-17T15:14:56.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-05-17T15:14:56.997 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-05-17T15:14:56.998 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr:2024-05-17T15:14:56.996+0000 7f1f07f75640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 46141) UID: 1000 2024-05-17T15:14:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_7': 2024-05-17T15:14:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:14:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:14:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:14:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d332701993 2024-05-17T15:14:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d332701993 2024-05-17T15:14:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:14:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:14:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:14:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:14:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:35 2024 2024-05-17T15:14:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:05:35 2024 2024-05-17T15:14:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:35 2024 2024-05-17T15:14:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:14:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:14:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:14:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 6e4b54d1-2cee-4ccb-aef6-8e1d0ba0e670 2024-05-17T15:14:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:14:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_7 snap 2024-05-17T15:14:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_7 2024-05-17T15:14:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:14:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_7.export 2024-05-17T15:14:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_7.export 2024-05-17T15:14:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_7.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_7.export 2024-05-17T15:14:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_7.export 2024-05-17T15:14:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:14:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_7.export 2024-05-17T15:14:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:14:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_7.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_7.export 2024-05-17T15:14:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_7.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_7.export 2024-05-17T15:14:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:14:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_8 2024-05-17T15:14:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_8 snap 2024-05-17T15:14:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:14:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:14:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:14:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-05-17T15:14:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:14:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-05-17T15:14:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:14:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:14:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:14:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:14:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:14:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-05-17T15:14:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:14:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:14:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:14:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:14:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:14:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-05-17T15:14:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:14:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:14:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:14:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:14:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:14:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:14:59.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:14:59.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:14:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:14:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:14:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-05-17T15:14:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:14:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:14:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:14:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:14:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:14:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:14:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:14:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:14:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:14:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:14:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:14:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:14:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:14:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:14:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:14:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:14:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:14:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:14:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:14:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:14:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:14:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:14:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:14:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:14:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:14:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:14:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:14:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:14:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:14:59.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:14:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-05-17T15:14:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:14:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:14:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:14:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:14:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:14:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:14:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:14:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:14:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-05-17T15:14:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:14:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:14:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:14:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:14:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2024-05-17T15:14:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:14:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:14:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_8.status 2024-05-17T15:14:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_8.status 2024-05-17T15:14:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:14:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_8.status 2024-05-17T15:14:59.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_8.status 2024-05-17T15:14:59.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_8.status 2024-05-17T15:14:59.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:14:59.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_8.status 2024-05-17T15:14:59.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-05-17T15:14:59.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:14:59.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:14:59.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:14:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-05-17T15:14:59.998 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-05-17T15:14:59.998 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:14:59.999 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-05-17T15:14:59.999 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-05-17T15:14:59.999 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-05-17T15:15:00.002 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-05-17T15:15:00.002 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-05-17T15:15:00.146 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2024-05-17T15:15:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 100 2024-05-17T15:15:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_8.status 2024-05-17T15:15:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:15:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_8.status 2024-05-17T15:15:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-05-17T15:15:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:15:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=100 2024-05-17T15:15:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:15:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_8.status 2024-05-17T15:15:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:15:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_8.status 2024-05-17T15:15:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-05-17T15:15:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=42 2024-05-17T15:15:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 100 = 42 2024-05-17T15:15:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:15:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_8.status 2024-05-17T15:15:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:15:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_8.status 2024-05-17T15:15:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-05-17T15:15:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:01.439 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:15:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:15:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:15:01.441 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:15:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 100 = '' 2024-05-17T15:15:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:01.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:15:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_8.status 2024-05-17T15:15:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:15:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_8.status 2024-05-17T15:15:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-05-17T15:15:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:15:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:15:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:15:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:15:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 100 = '' 2024-05-17T15:15:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:15:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_8.status 2024-05-17T15:15:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:15:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_8.status 2024-05-17T15:15:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-05-17T15:15:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=100 2024-05-17T15:15:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 100 = 100 2024-05-17T15:15:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-05-17T15:15:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:15:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:15:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:15:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:15:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-05-17T15:15:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_8': 2024-05-17T15:15:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:15:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:15:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:15:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 11351efa57ba 2024-05-17T15:15:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.11351efa57ba 2024-05-17T15:15:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:15:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:15:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:15:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:15:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:50 2024 2024-05-17T15:15:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:05:50 2024 2024-05-17T15:15:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:50 2024 2024-05-17T15:15:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:15:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:15:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:15:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 75a0868a-edaf-438e-a7e1-fedf558dccfa 2024-05-17T15:15:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:15:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_8 snap 2024-05-17T15:15:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_8 2024-05-17T15:15:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:15:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_8.export 2024-05-17T15:15:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_8.export 2024-05-17T15:15:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_8.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_8.export 2024-05-17T15:15:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_8.export 2024-05-17T15:15:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:15:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_8.export 2024-05-17T15:15:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:15:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_8.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_8.export 2024-05-17T15:15:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_8.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_8.export 2024-05-17T15:15:05.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~113s 2024-05-17T15:15:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:15:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_9 2024-05-17T15:15:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_9 snap 2024-05-17T15:15:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:15:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:05.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:15:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-05-17T15:15:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:15:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-05-17T15:15:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:15:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:15:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:15:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:15:06.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:15:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-05-17T15:15:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:15:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:15:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:15:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:15:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-05-17T15:15:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:15:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:15:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:15:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:15:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:15:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:15:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:15:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:15:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:15:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-05-17T15:15:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:15:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:15:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:15:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:15:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:15:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:15:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:15:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:15:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:15:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:15:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:15:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:15:07.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-05-17T15:15:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:15:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:15:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:15:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:15:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:15:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:15:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:15:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-05-17T15:15:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:15:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:15:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:15:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:15:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2024-05-17T15:15:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:15:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:15:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_9.status 2024-05-17T15:15:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_9.status 2024-05-17T15:15:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:15:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_9.status 2024-05-17T15:15:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_9.status 2024-05-17T15:15:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_9.status 2024-05-17T15:15:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:15:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_9.status 2024-05-17T15:15:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-05-17T15:15:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:15:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-05-17T15:15:07.993 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6930823452033786) > weight (0.25) 2024-05-17T15:15:07.993 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-05-17T15:15:07.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-05-17T15:15:07.993 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-05-17T15:15:07.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:15:07.992+0000 7fa8dc38b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 49241) UID: 1000 2024-05-17T15:15:08.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 102 2024-05-17T15:15:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_9.status 2024-05-17T15:15:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:15:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_9.status 2024-05-17T15:15:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-05-17T15:15:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:15:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=102 2024-05-17T15:15:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:15:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_9.status 2024-05-17T15:15:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:15:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_9.status 2024-05-17T15:15:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-05-17T15:15:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=43 2024-05-17T15:15:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 102 = 43 2024-05-17T15:15:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:09.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:15:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_9.status 2024-05-17T15:15:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:15:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_9.status 2024-05-17T15:15:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-05-17T15:15:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:15:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:15:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:15:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:15:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 102 = '' 2024-05-17T15:15:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:15:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_9.status 2024-05-17T15:15:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:15:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_9.status 2024-05-17T15:15:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-05-17T15:15:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:15:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:15:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:15:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:15:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 102 = '' 2024-05-17T15:15:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:15:10.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2024-05-17T15:15:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_9.status 2024-05-17T15:15:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:15:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_9.status 2024-05-17T15:15:11.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-05-17T15:15:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=102 2024-05-17T15:15:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 102 = 102 2024-05-17T15:15:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-05-17T15:15:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:15:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:15:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:15:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:15:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-05-17T15:15:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_9': 2024-05-17T15:15:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:15:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:15:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:15:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d349c640c3 2024-05-17T15:15:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d349c640c3 2024-05-17T15:15:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:15:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:15:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:15:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:15:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:59 2024 2024-05-17T15:15:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:05:59 2024 2024-05-17T15:15:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:59 2024 2024-05-17T15:15:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:15:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:15:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:15:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 6c3aab86-8ad4-4a10-8f48-3d66ec7f5b30 2024-05-17T15:15:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:15:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_9 snap 2024-05-17T15:15:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_9 2024-05-17T15:15:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:15:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_9.export 2024-05-17T15:15:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_9.export 2024-05-17T15:15:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_9.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_9.export 2024-05-17T15:15:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_9.export 2024-05-17T15:15:13.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:15:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_9.export 2024-05-17T15:15:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:15:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_9.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_9.export 2024-05-17T15:15:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_9.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_9.export 2024-05-17T15:15:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:15:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_10 2024-05-17T15:15:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_10 snap 2024-05-17T15:15:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:15:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:15:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-05-17T15:15:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:15:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-05-17T15:15:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:15:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:15:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:15:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:15:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:15:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-05-17T15:15:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:15:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:15:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:15:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:15:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-05-17T15:15:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:15:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:15:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:15:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:15:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:15:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:15:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:15:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:15:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:15:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-05-17T15:15:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:15:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:15:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:15:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:15:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:15:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:15:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:15:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:15:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:15:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:15:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:15:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:15:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-05-17T15:15:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-05-17T15:15:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-05-17T15:15:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:15:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:15:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:15:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:15:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:15:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:15:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:15:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:15:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:15:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:15:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:15:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:15:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:15:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:15:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:15:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:15:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-05-17T15:15:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-05-17T15:15:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2024-05-17T15:15:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:15:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:15:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:15:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:15:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:15:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:15:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:15:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:15:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:15:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:15:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:15:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:15:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:15:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:15:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:15:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:15:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:15:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:15:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:15:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2024-05-17T15:15:15.994 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-05-17T15:15:15.994 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:15:15.995 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-05-17T15:15:15.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-05-17T15:15:15.995 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-05-17T15:15:15.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-05-17T15:15:15.998 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2024-05-17T15:15:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-05-17T15:15:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_10 2024-05-17T15:15:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:15:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:15:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:15:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:15:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:15:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:15:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:15:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:15:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:15:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:15:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:15:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:15:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:15:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:15:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:15:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:15:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:15:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:15:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:15:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_10 2024-05-17T15:15:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:15:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:15:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:15:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:15:16.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:15:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:15:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:15:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-05-17T15:15:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:15:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:15:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:15:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:15:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2024-05-17T15:15:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:15:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:15:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_10.status 2024-05-17T15:15:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_10.status 2024-05-17T15:15:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:15:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_10.status 2024-05-17T15:15:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_10.status 2024-05-17T15:15:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_10.status 2024-05-17T15:15:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:15:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_10.status 2024-05-17T15:15:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-05-17T15:15:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:15:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-05-17T15:15:16.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2024-05-17T15:15:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 104 2024-05-17T15:15:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_10.status 2024-05-17T15:15:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:15:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_10.status 2024-05-17T15:15:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-05-17T15:15:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:15:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=104 2024-05-17T15:15:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:15:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_10.status 2024-05-17T15:15:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:15:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_10.status 2024-05-17T15:15:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-05-17T15:15:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=44 2024-05-17T15:15:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 104 = 44 2024-05-17T15:15:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:15:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_10.status 2024-05-17T15:15:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:15:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_10.status 2024-05-17T15:15:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-05-17T15:15:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:15:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:15:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:15:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:15:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 104 = '' 2024-05-17T15:15:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:15:18.308 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_10.status 2024-05-17T15:15:18.308 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:18.308 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:15:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_10.status 2024-05-17T15:15:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-05-17T15:15:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:18.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:15:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:15:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:15:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:15:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 104 = '' 2024-05-17T15:15:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:15:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_10.status 2024-05-17T15:15:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:15:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_10.status 2024-05-17T15:15:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-05-17T15:15:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=104 2024-05-17T15:15:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 104 = 104 2024-05-17T15:15:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-05-17T15:15:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:15:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:15:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:15:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:15:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-05-17T15:15:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_10': 2024-05-17T15:15:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:15:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:15:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:15:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1135c02e8df3 2024-05-17T15:15:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1135c02e8df3 2024-05-17T15:15:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:15:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:15:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:15:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:15:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:06:11 2024 2024-05-17T15:15:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:06:11 2024 2024-05-17T15:15:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:06:11 2024 2024-05-17T15:15:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:15:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:15:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:15:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 95950a54-837a-461f-aafc-36eac3b2a17a 2024-05-17T15:15:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:15:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_10 snap 2024-05-17T15:15:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_10 2024-05-17T15:15:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:15:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_10.export 2024-05-17T15:15:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_10.export 2024-05-17T15:15:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_10.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_10.export 2024-05-17T15:15:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_10.export 2024-05-17T15:15:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:15:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_10.export 2024-05-17T15:15:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:15:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_10.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_10.export 2024-05-17T15:15:21.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_10.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_10.export 2024-05-17T15:15:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:15:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_11 2024-05-17T15:15:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_11 snap 2024-05-17T15:15:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:15:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:15:21.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2024-05-17T15:15:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-05-17T15:15:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:15:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-05-17T15:15:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:15:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:15:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:15:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:15:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:15:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-05-17T15:15:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:15:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:15:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:15:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:15:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-05-17T15:15:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:15:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:15:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:15:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:15:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:15:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:15:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:15:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:15:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:15:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-05-17T15:15:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:15:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:15:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:15:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:15:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:15:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:15:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:15:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:15:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:15:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:15:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:15:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:15:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-05-17T15:15:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-05-17T15:15:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-05-17T15:15:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:15:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:15:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:15:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:15:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:15:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:15:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:15:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:15:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:15:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:15:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:15:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:15:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:15:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:15:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:15:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:15:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-05-17T15:15:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-05-17T15:15:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2024-05-17T15:15:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:15:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:15:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:15:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:23.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:15:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:15:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:15:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:15:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:15:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:15:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:15:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:15:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2024-05-17T15:15:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-05-17T15:15:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_11 2024-05-17T15:15:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:15:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:15:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:15:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:15:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:15:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:15:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:15:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:15:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:15:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:15:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:15:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:15:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:15:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:15:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:15:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:15:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:15:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:15:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:15:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_11 2024-05-17T15:15:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-05-17T15:15:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:24.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:15:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 1 2024-05-17T15:15:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:15:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:15:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-05-17T15:15:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:15:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:15:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:15:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:15:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:15:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:15:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:15:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:15:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:15:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:15:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:15:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:15:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-05-17T15:15:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-05-17T15:15:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-05-17T15:15:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:15:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:15:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:15:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:15:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:15:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:15:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:15:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:15:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:15:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:15:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:15:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:15:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:15:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:15:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:15:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:15:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-05-17T15:15:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-05-17T15:15:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2024-05-17T15:15:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:15:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:15:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:15:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:15:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:15:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:15:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:15:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:15:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:15:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:15:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:15:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:15:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:15:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:15:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:15:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:15:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:15:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:15:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:15:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2024-05-17T15:15:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-05-17T15:15:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_11 2024-05-17T15:15:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:15:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:15:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:15:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:15:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:15:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:15:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:15:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:15:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:15:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:15:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:15:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:15:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:15:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:15:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:15:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:15:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:15:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:15:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:15:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_11 2024-05-17T15:15:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-05-17T15:15:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:15:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 2 2024-05-17T15:15:27.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2024-05-17T15:15:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:15:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:15:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-05-17T15:15:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:15:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:15:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:15:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:15:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:15:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:15:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:15:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:15:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:15:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:15:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:15:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:15:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-05-17T15:15:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-05-17T15:15:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-05-17T15:15:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:15:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:15:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:15:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:15:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:15:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:15:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:15:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:15:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:15:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:15:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:15:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:15:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:15:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:15:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:15:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:15:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-05-17T15:15:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-05-17T15:15:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2024-05-17T15:15:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:15:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:15:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:15:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:15:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:15:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:15:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:15:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:15:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:15:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:15:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:15:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:15:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:15:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:15:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:15:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:15:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:15:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:15:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:15:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2024-05-17T15:15:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-05-17T15:15:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_11 2024-05-17T15:15:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:15:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:15:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:15:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:15:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:15:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:15:27.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:15:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:15:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:15:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:15:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:15:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:15:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:15:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:15:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:15:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:15:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:15:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:15:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:15:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_11 2024-05-17T15:15:28.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-05-17T15:15:28.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:28.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:15:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 4 2024-05-17T15:15:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:15:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:15:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-05-17T15:15:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:15:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:15:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:15:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:15:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:15:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:15:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:15:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:15:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:15:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:15:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:15:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:15:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-05-17T15:15:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-05-17T15:15:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-05-17T15:15:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:15:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:32.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:15:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:15:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:15:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:15:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:15:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:15:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:15:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:15:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:15:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:15:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:15:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:15:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:15:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:15:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:15:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-05-17T15:15:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-05-17T15:15:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2024-05-17T15:15:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:15:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:15:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:15:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:15:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:15:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:15:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:15:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:15:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:15:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:15:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:15:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:15:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:15:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:15:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:15:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:15:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:15:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:15:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:15:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2024-05-17T15:15:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-05-17T15:15:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_11 2024-05-17T15:15:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:15:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:15:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:15:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:15:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:15:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:15:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:15:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:15:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:15:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:15:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:15:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:15:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:15:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:15:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:15:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:15:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:15:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:15:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:15:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_11 2024-05-17T15:15:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-05-17T15:15:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:15:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 8 2024-05-17T15:15:32.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~141s 2024-05-17T15:15:37.975 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2024-05-17T15:15:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:15:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:15:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-05-17T15:15:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:15:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:15:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:15:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:15:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:15:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:15:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:15:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:15:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:15:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:15:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:15:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:15:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-05-17T15:15:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-05-17T15:15:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-05-17T15:15:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:15:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:15:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:15:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:15:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:15:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:15:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:15:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:15:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:15:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:15:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:15:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:15:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:15:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:15:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:15:40.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:15:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:40.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:40.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:40.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-05-17T15:15:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:15:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:15:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:15:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:15:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:15:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:15:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:15:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-05-17T15:15:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:15:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:15:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:15:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:15:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2024-05-17T15:15:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:15:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:15:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_11.status 2024-05-17T15:15:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_11.status 2024-05-17T15:15:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:15:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_11.status 2024-05-17T15:15:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_11.status 2024-05-17T15:15:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_11.status 2024-05-17T15:15:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:15:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_11.status 2024-05-17T15:15:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-05-17T15:15:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:40.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:40.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:15:40.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-05-17T15:15:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 106 2024-05-17T15:15:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_11.status 2024-05-17T15:15:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:15:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_11.status 2024-05-17T15:15:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-05-17T15:15:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:15:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=106 2024-05-17T15:15:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:15:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_11.status 2024-05-17T15:15:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:15:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_11.status 2024-05-17T15:15:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-05-17T15:15:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:15:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:15:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:15:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:15:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 106 = '' 2024-05-17T15:15:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:15:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_11.status 2024-05-17T15:15:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:15:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_11.status 2024-05-17T15:15:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-05-17T15:15:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:15:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:15:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:15:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:15:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 106 = '' 2024-05-17T15:15:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:15:42.236 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:15:42.234+0000 7f7a4dbdb640 -1 reset not still connected to 0x55d8fac4a000 2024-05-17T15:15:42.237 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:15:42.234+0000 7f7a4dbdb640 -1 reset not still connected to 0x55d8fd5dd110 2024-05-17T15:15:42.237 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:15:42.234+0000 7f7a4dbdb640 -1 reset not still connected to 0x55d90131e680 2024-05-17T15:15:42.237 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:15:42.234+0000 7f7a4dbdb640 -1 reset not still connected to 0x55d9046b1ad0 2024-05-17T15:15:42.237 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:15:42.234+0000 7f7a4dbdb640 -1 reset not still connected to 0x55d912aaf930 2024-05-17T15:15:42.237 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:15:42.234+0000 7f7a4dbdb640 -1 reset not still connected to 0x55d91f252820 2024-05-17T15:15:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_11.status 2024-05-17T15:15:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:15:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_11.status 2024-05-17T15:15:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-05-17T15:15:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:15:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:15:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:15:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:15:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 106 = '' 2024-05-17T15:15:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:15:43.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2024-05-17T15:15:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_11.status 2024-05-17T15:15:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:15:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_11.status 2024-05-17T15:15:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-05-17T15:15:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=106 2024-05-17T15:15:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 106 = 106 2024-05-17T15:15:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-05-17T15:15:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:15:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:15:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:15:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:15:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-05-17T15:15:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_11': 2024-05-17T15:15:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:15:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:15:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:15:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 121ddc84b4e9 2024-05-17T15:15:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.121ddc84b4e9 2024-05-17T15:15:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:15:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:15:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:15:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:15:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:06:17 2024 2024-05-17T15:15:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:06:17 2024 2024-05-17T15:15:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:06:17 2024 2024-05-17T15:15:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:15:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:15:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:15:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 0e1e48a7-bf0e-42d2-9d09-07eeeeaaf56e 2024-05-17T15:15:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:15:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_11 snap 2024-05-17T15:15:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_11 2024-05-17T15:15:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:15:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_11.export 2024-05-17T15:15:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_11.export 2024-05-17T15:15:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_11.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_11.export 2024-05-17T15:15:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_11.export 2024-05-17T15:15:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:15:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_11.export 2024-05-17T15:15:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:15:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_11.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_11.export 2024-05-17T15:15:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_11.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_11.export 2024-05-17T15:15:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:15:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_12 2024-05-17T15:15:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_12 snap 2024-05-17T15:15:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_12 2024-05-17T15:15:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:15:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-05-17T15:15:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_12 2024-05-17T15:15:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-05-17T15:15:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_12 2024-05-17T15:15:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:15:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:15:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:15:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:15:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-05-17T15:15:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_12 2024-05-17T15:15:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:15:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:15:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:15:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-05-17T15:15:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:15:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:15:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:15:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:15:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:15:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:15:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:15:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:15:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:15:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-05-17T15:15:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:15:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:15:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:15:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:15:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:15:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:15:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:15:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:15:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:15:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:15:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:15:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:15:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-05-17T15:15:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-05-17T15:15:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:15:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:15:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-05-17T15:15:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:15:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:15:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:15:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:15:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:15:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:15:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:15:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:15:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:15:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:15:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:15:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:15:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:15:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:15:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:15:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:15:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:15:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-05-17T15:15:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:15:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:15:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:15:48.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:15:48.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:15:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:15:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:15:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-05-17T15:15:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:15:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_12 2024-05-17T15:15:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:15:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:15:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2024-05-17T15:15:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:15:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_12 2024-05-17T15:15:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_12.status 2024-05-17T15:15:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_12.status 2024-05-17T15:15:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:15:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_12.status 2024-05-17T15:15:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_12.status 2024-05-17T15:15:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_12.status 2024-05-17T15:15:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:15:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_12.status 2024-05-17T15:15:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-05-17T15:15:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_12 2024-05-17T15:15:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-05-17T15:15:48.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2024-05-17T15:15:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 108 2024-05-17T15:15:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_12.status 2024-05-17T15:15:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_12 2024-05-17T15:15:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_12.status 2024-05-17T15:15:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-05-17T15:15:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:15:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=108 2024-05-17T15:15:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:15:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_12.status 2024-05-17T15:15:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_12 2024-05-17T15:15:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_12.status 2024-05-17T15:15:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-05-17T15:15:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:49.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:49.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:15:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:15:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:15:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:15:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 108 = '' 2024-05-17T15:15:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:15:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_12.status 2024-05-17T15:15:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_12 2024-05-17T15:15:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_12.status 2024-05-17T15:15:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-05-17T15:15:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:15:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:15:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:15:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:15:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 108 = '' 2024-05-17T15:15:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:15:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_12.status 2024-05-17T15:15:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_12 2024-05-17T15:15:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_12.status 2024-05-17T15:15:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-05-17T15:15:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=108 2024-05-17T15:15:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 108 = 108 2024-05-17T15:15:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-05-17T15:15:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_12 2024-05-17T15:15:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:15:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:15:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:15:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-05-17T15:15:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_12': 2024-05-17T15:15:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:15:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:15:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:15:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1284a2612bb4 2024-05-17T15:15:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1284a2612bb4 2024-05-17T15:15:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:15:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:15:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:15:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:15:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:06:31 2024 2024-05-17T15:15:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:06:31 2024 2024-05-17T15:15:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:06:31 2024 2024-05-17T15:15:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:15:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:15:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:15:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: c50f08e5-ef2e-4cc9-afd7-fb18dd3f03dc 2024-05-17T15:15:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:15:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_12 snap 2024-05-17T15:15:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_12 2024-05-17T15:15:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:15:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_12.export 2024-05-17T15:15:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_12.export 2024-05-17T15:15:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_12.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_12.export 2024-05-17T15:15:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_12.export 2024-05-17T15:15:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:15:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_12.export 2024-05-17T15:15:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:15:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_12.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_12.export 2024-05-17T15:15:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_12.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_12.export 2024-05-17T15:15:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:15:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_13 2024-05-17T15:15:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_13 snap 2024-05-17T15:15:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_13 2024-05-17T15:15:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:15:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-05-17T15:15:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_13 2024-05-17T15:15:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-05-17T15:15:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_13 2024-05-17T15:15:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:15:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:15:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:15:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:15:54.188 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2024-05-17T15:15:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-05-17T15:15:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_13 2024-05-17T15:15:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:15:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:15:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:15:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-05-17T15:15:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:15:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:15:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:15:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:15:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:15:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:15:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:15:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:15:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:15:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:15:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-05-17T15:15:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:15:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:15:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:15:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:15:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:15:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:15:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:15:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:15:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:15:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:15:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:15:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:15:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:15:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:15:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:15:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:15:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:15:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:15:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:15:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:15:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:15:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:15:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:15:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:15:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:15:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:15:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:15:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:15:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-05-17T15:15:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:15:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:15:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:15:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:15:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:15:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-05-17T15:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_13 2024-05-17T15:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2024-05-17T15:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_13 2024-05-17T15:15:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_13.status 2024-05-17T15:15:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_13.status 2024-05-17T15:15:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_13.status 2024-05-17T15:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_13.status 2024-05-17T15:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_13.status 2024-05-17T15:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_13.status 2024-05-17T15:15:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-05-17T15:15:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_13 2024-05-17T15:15:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-05-17T15:15:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 110 2024-05-17T15:15:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_13.status 2024-05-17T15:15:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_13 2024-05-17T15:15:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_13.status 2024-05-17T15:15:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-05-17T15:15:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:55.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:15:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=110 2024-05-17T15:15:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:15:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_13.status 2024-05-17T15:15:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_13 2024-05-17T15:15:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_13.status 2024-05-17T15:15:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-05-17T15:15:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:15:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:15:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:15:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:15:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 110 = '' 2024-05-17T15:15:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:15:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_13.status 2024-05-17T15:15:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_13 2024-05-17T15:15:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_13.status 2024-05-17T15:15:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-05-17T15:15:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:56.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:15:56.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:15:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:15:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:15:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:15:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 110 = '' 2024-05-17T15:15:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:15:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:15:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_13.status 2024-05-17T15:15:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_13 2024-05-17T15:15:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_13.status 2024-05-17T15:15:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-05-17T15:15:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:15:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:15:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=110 2024-05-17T15:15:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 110 = 110 2024-05-17T15:15:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-05-17T15:15:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:15:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_13 2024-05-17T15:15:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:15:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:15:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:15:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-05-17T15:15:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_13': 2024-05-17T15:15:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:15:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:15:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:15:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d3d5590995 2024-05-17T15:15:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d3d5590995 2024-05-17T15:15:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:15:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:15:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:15:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:15:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:06:38 2024 2024-05-17T15:15:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:06:38 2024 2024-05-17T15:15:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:06:38 2024 2024-05-17T15:15:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:15:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:15:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:15:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 76d62c45-9c33-4b38-aeb9-d12dc0c2247a 2024-05-17T15:15:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:15:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_13 snap 2024-05-17T15:15:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_13 2024-05-17T15:15:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:15:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_13.export 2024-05-17T15:15:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_13.export 2024-05-17T15:15:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_13.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_13.export 2024-05-17T15:15:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_13.export 2024-05-17T15:15:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:15:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_13.export 2024-05-17T15:15:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:15:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_13.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_13.export 2024-05-17T15:15:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_13.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_13.export 2024-05-17T15:15:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:15:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:15:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_14 2024-05-17T15:15:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_14 snap 2024-05-17T15:15:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:15:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:15:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_14 2024-05-17T15:15:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:15:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:15:59.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2024-05-17T15:16:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-05-17T15:16:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_14 2024-05-17T15:16:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-05-17T15:16:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_14 2024-05-17T15:16:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:16:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:16:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:16:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-05-17T15:16:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_14 2024-05-17T15:16:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:16:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:16:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:16:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-05-17T15:16:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:16:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:16:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:16:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:16:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:16:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:16:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:16:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:16:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-05-17T15:16:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:16:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:16:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:16:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:16:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:16:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:16:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:16:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:16:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:16:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:16:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:16:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:16:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-05-17T15:16:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-05-17T15:16:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-05-17T15:16:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:16:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:16:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:16:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:16:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:16:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:16:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:16:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:16:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:16:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:16:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:16:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:16:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-05-17T15:16:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:16:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:16:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:16:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:16:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-05-17T15:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_14 2024-05-17T15:16:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2024-05-17T15:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_14 2024-05-17T15:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_14.status 2024-05-17T15:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_14.status 2024-05-17T15:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_14.status 2024-05-17T15:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_14.status 2024-05-17T15:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_14.status 2024-05-17T15:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_14.status 2024-05-17T15:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-05-17T15:16:02.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_14 2024-05-17T15:16:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-05-17T15:16:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 112 2024-05-17T15:16:02.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_14.status 2024-05-17T15:16:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_14 2024-05-17T15:16:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_14.status 2024-05-17T15:16:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-05-17T15:16:02.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:02.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:16:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=112 2024-05-17T15:16:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:16:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:16:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_14.status 2024-05-17T15:16:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_14 2024-05-17T15:16:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_14.status 2024-05-17T15:16:03.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-05-17T15:16:03.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:16:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:16:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:16:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:16:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:16:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 112 = '' 2024-05-17T15:16:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:16:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_14.status 2024-05-17T15:16:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_14 2024-05-17T15:16:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_14.status 2024-05-17T15:16:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-05-17T15:16:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:16:03.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:16:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:16:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:16:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:16:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 112 = '' 2024-05-17T15:16:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:16:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_14.status 2024-05-17T15:16:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_14 2024-05-17T15:16:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_14.status 2024-05-17T15:16:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-05-17T15:16:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:04.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=112 2024-05-17T15:16:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 112 = 112 2024-05-17T15:16:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-05-17T15:16:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_14 2024-05-17T15:16:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:16:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:16:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:16:04.996 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2024-05-17T15:16:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-05-17T15:16:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_14': 2024-05-17T15:16:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:16:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:16:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:16:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1135b7eb2bb 2024-05-17T15:16:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1135b7eb2bb 2024-05-17T15:16:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:16:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:16:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:16:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:16:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:06:49 2024 2024-05-17T15:16:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:06:49 2024 2024-05-17T15:16:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:06:49 2024 2024-05-17T15:16:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:16:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:16:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:16:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 9e41e805-f938-4777-a0dd-ce32b1acfcf1 2024-05-17T15:16:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:16:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_14 snap 2024-05-17T15:16:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_14 2024-05-17T15:16:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:16:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_14.export 2024-05-17T15:16:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_14.export 2024-05-17T15:16:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_14.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_14.export 2024-05-17T15:16:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_14.export 2024-05-17T15:16:05.777 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:16:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_14.export 2024-05-17T15:16:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:16:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_14.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_14.export 2024-05-17T15:16:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_14.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_14.export 2024-05-17T15:16:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:16:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_15 2024-05-17T15:16:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_15 snap 2024-05-17T15:16:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:16:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:16:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-05-17T15:16:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:16:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-05-17T15:16:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:16:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:16:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:16:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:16:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-05-17T15:16:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:16:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:16:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:16:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-05-17T15:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:16:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:16:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-05-17T15:16:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:16:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:16:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:16:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:16:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:16:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:16:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:16:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:16:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:16:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:16:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:16:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:16:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-05-17T15:16:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-05-17T15:16:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2024-05-17T15:16:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:16:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:16:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:16:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:16:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:16:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:16:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:16:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:16:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:16:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:16:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:16:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:16:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2024-05-17T15:16:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-05-17T15:16:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_15 2024-05-17T15:16:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:16:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:16:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:16:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:16:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:16:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:16:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:16:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:16:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:16:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:16:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:16:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:16:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:16:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:16:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:16:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:16:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_15 2024-05-17T15:16:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-05-17T15:16:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_15 2024-05-17T15:16:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:16:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:16:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:16:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:16:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:16:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:16:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:16:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:16:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:16:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:16:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:16:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:16:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:16:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:16:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:16:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:16:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:16:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:16:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:16:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_15 2024-05-17T15:16:08.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:16:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:16:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:16:08.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:16:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:16:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:16:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:16:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-05-17T15:16:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:16:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:16:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:16:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:16:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2024-05-17T15:16:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:16:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:16:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_15.status 2024-05-17T15:16:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_15.status 2024-05-17T15:16:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:16:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_15.status 2024-05-17T15:16:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_15.status 2024-05-17T15:16:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_15.status 2024-05-17T15:16:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:16:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_15.status 2024-05-17T15:16:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-05-17T15:16:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:16:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-05-17T15:16:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 114 2024-05-17T15:16:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_15.status 2024-05-17T15:16:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:16:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_15.status 2024-05-17T15:16:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-05-17T15:16:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:16:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=114 2024-05-17T15:16:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:16:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:16:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_15.status 2024-05-17T15:16:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:16:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_15.status 2024-05-17T15:16:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-05-17T15:16:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=49 2024-05-17T15:16:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 114 = 49 2024-05-17T15:16:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:16:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_15.status 2024-05-17T15:16:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:16:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_15.status 2024-05-17T15:16:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-05-17T15:16:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=49 2024-05-17T15:16:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 114 = 49 2024-05-17T15:16:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:16:10.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2024-05-17T15:16:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_15.status 2024-05-17T15:16:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:16:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_15.status 2024-05-17T15:16:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-05-17T15:16:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:16:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:16:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:16:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:16:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:16:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 114 = '' 2024-05-17T15:16:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:16:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_15.status 2024-05-17T15:16:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:16:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_15.status 2024-05-17T15:16:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-05-17T15:16:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=114 2024-05-17T15:16:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 114 = 114 2024-05-17T15:16:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-05-17T15:16:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:16:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:16:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:16:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:16:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-05-17T15:16:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_15': 2024-05-17T15:16:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:16:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:16:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:16:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1293dc4d5196 2024-05-17T15:16:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1293dc4d5196 2024-05-17T15:16:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:16:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:16:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:16:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:16:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:07:22 2024 2024-05-17T15:16:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:07:22 2024 2024-05-17T15:16:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:07:22 2024 2024-05-17T15:16:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:16:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:16:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:16:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: fb0ce489-3f67-44e1-971c-03b1005074a7 2024-05-17T15:16:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:16:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_15 snap 2024-05-17T15:16:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_15 2024-05-17T15:16:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:16:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_15.export 2024-05-17T15:16:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_15.export 2024-05-17T15:16:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_15.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_15.export 2024-05-17T15:16:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_15.export 2024-05-17T15:16:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:16:13.803 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_15.export 2024-05-17T15:16:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:16:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_15.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_15.export 2024-05-17T15:16:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_15.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_15.export 2024-05-17T15:16:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:14.141 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:16:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_16 2024-05-17T15:16:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_16 snap 2024-05-17T15:16:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:16:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:16:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-05-17T15:16:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:16:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-05-17T15:16:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:16:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:16:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:16:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:16:15.804 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2024-05-17T15:16:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-05-17T15:16:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:16:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:16:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:16:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:16:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-05-17T15:16:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:16:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:16:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:16:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:16:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:16:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:16:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:16:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:16:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-05-17T15:16:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:16:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:16:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:16:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:16:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:16:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:16:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:16:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-05-17T15:16:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-05-17T15:16:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-05-17T15:16:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:16:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:16:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:16:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:16:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:16:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:16:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-05-17T15:16:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-05-17T15:16:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2024-05-17T15:16:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:16:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:16:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:16:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:16:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:16:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:16:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:16:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:16:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:16:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:16:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:16:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:16:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:16:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:16:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:16:16.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:16:16.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:16.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:16.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:16.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2024-05-17T15:16:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:16:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:16:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:16:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:16:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:16:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:16:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:16:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-05-17T15:16:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:16:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:16:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:16:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:16:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2024-05-17T15:16:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:16:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:16:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_16.status 2024-05-17T15:16:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_16.status 2024-05-17T15:16:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:16:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_16.status 2024-05-17T15:16:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_16.status 2024-05-17T15:16:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_16.status 2024-05-17T15:16:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:16:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_16.status 2024-05-17T15:16:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-05-17T15:16:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:16:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-05-17T15:16:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 116 2024-05-17T15:16:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_16.status 2024-05-17T15:16:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:16:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_16.status 2024-05-17T15:16:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-05-17T15:16:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=116 2024-05-17T15:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:16:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:16:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_16.status 2024-05-17T15:16:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:16:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_16.status 2024-05-17T15:16:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-05-17T15:16:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=50 2024-05-17T15:16:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 116 = 50 2024-05-17T15:16:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:16:18.003 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-05-17T15:16:18.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-05-17T15:16:18.003 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-05-17T15:16:18.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr:2024-05-17T15:16:18.002+0000 7f3056595640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 53896) UID: 1000 2024-05-17T15:16:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_16.status 2024-05-17T15:16:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:16:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_16.status 2024-05-17T15:16:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-05-17T15:16:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:16:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:16:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:16:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:16:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:16:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 116 = '' 2024-05-17T15:16:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:16:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_16.status 2024-05-17T15:16:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:16:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_16.status 2024-05-17T15:16:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-05-17T15:16:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:16:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:16:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:16:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:16:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:16:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 116 = '' 2024-05-17T15:16:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:16:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_16.status 2024-05-17T15:16:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:16:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_16.status 2024-05-17T15:16:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-05-17T15:16:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=116 2024-05-17T15:16:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 116 = 116 2024-05-17T15:16:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-05-17T15:16:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:16:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:16:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:16:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:16:21.004 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-05-17T15:16:21.005 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:16:21.005 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-05-17T15:16:21.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-05-17T15:16:21.005 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-05-17T15:16:21.008 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-05-17T15:16:21.008 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-05-17T15:16:21.168 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:16:21.166+0000 7f7a4dbdb640 -1 reset not still connected to 0x55d8eb74cd00 2024-05-17T15:16:21.168 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:16:21.166+0000 7f7a4dbdb640 -1 reset not still connected to 0x55d8fcb79790 2024-05-17T15:16:21.169 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:16:21.166+0000 7f7a4dbdb640 -1 reset not still connected to 0x55d9232731e0 2024-05-17T15:16:21.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2024-05-17T15:16:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-05-17T15:16:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_16': 2024-05-17T15:16:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:16:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:16:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:16:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1293779945bc 2024-05-17T15:16:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1293779945bc 2024-05-17T15:16:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:16:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:16:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:16:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:16:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:07:10 2024 2024-05-17T15:16:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:07:10 2024 2024-05-17T15:16:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:07:10 2024 2024-05-17T15:16:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:16:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:16:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:16:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 9cc94201-e324-4319-a646-d571d0e3df05 2024-05-17T15:16:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:16:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_16 snap 2024-05-17T15:16:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_16 2024-05-17T15:16:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:16:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_16.export 2024-05-17T15:16:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_16.export 2024-05-17T15:16:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_16.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_16.export 2024-05-17T15:16:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_16.export 2024-05-17T15:16:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:16:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_16.export 2024-05-17T15:16:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:16:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_16.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_16.export 2024-05-17T15:16:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_16.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_16.export 2024-05-17T15:16:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:16:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_17 2024-05-17T15:16:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_17 snap 2024-05-17T15:16:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:16:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:16:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-05-17T15:16:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:16:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-05-17T15:16:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:16:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:16:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:16:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:16:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-05-17T15:16:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:16:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:16:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:16:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:16:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-05-17T15:16:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:16:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:16:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:16:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:16:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:16:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:16:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:16:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:16:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-05-17T15:16:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:16:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:16:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:16:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:16:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:16:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:16:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:16:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:16:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:16:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:16:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:16:24.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:16:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-05-17T15:16:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-05-17T15:16:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2024-05-17T15:16:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:16:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:16:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:16:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:16:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:16:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:16:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:16:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:16:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:16:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:16:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:16:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:16:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2024-05-17T15:16:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:16:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:16:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:16:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:16:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:16:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:16:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:16:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-05-17T15:16:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:16:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:16:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:16:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:16:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2024-05-17T15:16:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:16:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:16:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_17.status 2024-05-17T15:16:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_17.status 2024-05-17T15:16:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:16:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_17.status 2024-05-17T15:16:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_17.status 2024-05-17T15:16:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_17.status 2024-05-17T15:16:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:16:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_17.status 2024-05-17T15:16:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-05-17T15:16:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:16:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-05-17T15:16:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 118 2024-05-17T15:16:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_17.status 2024-05-17T15:16:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:16:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_17.status 2024-05-17T15:16:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-05-17T15:16:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:16:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=118 2024-05-17T15:16:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:16:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:16:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_17.status 2024-05-17T15:16:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:16:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_17.status 2024-05-17T15:16:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-05-17T15:16:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=51 2024-05-17T15:16:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 118 = 51 2024-05-17T15:16:25.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:25.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:16:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_17.status 2024-05-17T15:16:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:16:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_17.status 2024-05-17T15:16:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-05-17T15:16:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:16:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:16:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:16:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:16:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:16:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 118 = '' 2024-05-17T15:16:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:16:26.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2024-05-17T15:16:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_17.status 2024-05-17T15:16:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:16:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_17.status 2024-05-17T15:16:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-05-17T15:16:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:16:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:16:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:16:27.277 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:16:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:16:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 118 = '' 2024-05-17T15:16:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:16:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_17.status 2024-05-17T15:16:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:16:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_17.status 2024-05-17T15:16:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-05-17T15:16:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=118 2024-05-17T15:16:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 118 = 118 2024-05-17T15:16:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-05-17T15:16:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:16:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:16:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:16:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:16:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-05-17T15:16:29.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_17': 2024-05-17T15:16:29.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:16:29.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:16:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:16:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d31974d5da 2024-05-17T15:16:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d31974d5da 2024-05-17T15:16:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:16:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:16:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:16:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:16:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:07:23 2024 2024-05-17T15:16:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:07:23 2024 2024-05-17T15:16:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:07:23 2024 2024-05-17T15:16:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:16:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:16:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:16:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: bef15777-4cd7-47fb-8829-12dabc805b90 2024-05-17T15:16:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:16:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_17 snap 2024-05-17T15:16:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_17 2024-05-17T15:16:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:16:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_17.export 2024-05-17T15:16:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_17.export 2024-05-17T15:16:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_17.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_17.export 2024-05-17T15:16:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_17.export 2024-05-17T15:16:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:16:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_17.export 2024-05-17T15:16:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:16:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_17.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_17.export 2024-05-17T15:16:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_17.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_17.export 2024-05-17T15:16:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:16:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_18 2024-05-17T15:16:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_18 snap 2024-05-17T15:16:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:16:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:16:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-05-17T15:16:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:16:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-05-17T15:16:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:16:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:16:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:16:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:16:32.016 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2024-05-17T15:16:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-05-17T15:16:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:16:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:16:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:16:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:16:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-05-17T15:16:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:16:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:16:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:16:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:16:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:16:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:16:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:16:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:16:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-05-17T15:16:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:16:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:16:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:16:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:16:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:16:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:16:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:16:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:16:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:16:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:16:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:16:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:16:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-05-17T15:16:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-05-17T15:16:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-05-17T15:16:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:16:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:16:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:16:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:16:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:16:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:16:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:33.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-05-17T15:16:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-05-17T15:16:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2024-05-17T15:16:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:16:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:16:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:16:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:16:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:16:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:16:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:16:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:16:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:16:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:16:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:16:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:16:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:16:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:16:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:16:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:16:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2024-05-17T15:16:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:16:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:16:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:16:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:16:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:16:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:16:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:16:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-05-17T15:16:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:16:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:16:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:16:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:16:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2024-05-17T15:16:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:16:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:16:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_18.status 2024-05-17T15:16:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_18.status 2024-05-17T15:16:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:16:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_18.status 2024-05-17T15:16:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_18.status 2024-05-17T15:16:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_18.status 2024-05-17T15:16:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:16:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_18.status 2024-05-17T15:16:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-05-17T15:16:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:16:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-05-17T15:16:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 120 2024-05-17T15:16:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_18.status 2024-05-17T15:16:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:16:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_18.status 2024-05-17T15:16:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-05-17T15:16:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:16:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=120 2024-05-17T15:16:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:16:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:16:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_18.status 2024-05-17T15:16:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:16:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_18.status 2024-05-17T15:16:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-05-17T15:16:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=52 2024-05-17T15:16:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 120 = 52 2024-05-17T15:16:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:16:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_18.status 2024-05-17T15:16:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:16:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_18.status 2024-05-17T15:16:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-05-17T15:16:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:16:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:16:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:16:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:16:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:16:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 120 = '' 2024-05-17T15:16:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:16:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_18.status 2024-05-17T15:16:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:16:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_18.status 2024-05-17T15:16:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-05-17T15:16:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:35.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:16:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:16:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:16:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:16:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:16:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 120 = '' 2024-05-17T15:16:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:16:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_18.status 2024-05-17T15:16:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:16:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_18.status 2024-05-17T15:16:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-05-17T15:16:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=120 2024-05-17T15:16:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 120 = 120 2024-05-17T15:16:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-05-17T15:16:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:16:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:16:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:16:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:16:37.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2024-05-17T15:16:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-05-17T15:16:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_18': 2024-05-17T15:16:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:16:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:16:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:16:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1135c4c03b23 2024-05-17T15:16:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1135c4c03b23 2024-05-17T15:16:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:16:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:16:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:16:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:16:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:07:38 2024 2024-05-17T15:16:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:07:38 2024 2024-05-17T15:16:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:07:38 2024 2024-05-17T15:16:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:16:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:16:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:16:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: e3330572-d02a-4104-a14f-d83973929a2c 2024-05-17T15:16:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:16:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_18 snap 2024-05-17T15:16:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_18 2024-05-17T15:16:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:16:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_18.export 2024-05-17T15:16:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_18.export 2024-05-17T15:16:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_18.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_18.export 2024-05-17T15:16:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_18.export 2024-05-17T15:16:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:16:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_18.export 2024-05-17T15:16:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:16:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_18.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_18.export 2024-05-17T15:16:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_18.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_18.export 2024-05-17T15:16:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:16:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_19 2024-05-17T15:16:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_19 snap 2024-05-17T15:16:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_19 2024-05-17T15:16:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:38.693 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:16:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-05-17T15:16:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_19 2024-05-17T15:16:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-05-17T15:16:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_19 2024-05-17T15:16:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:16:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:16:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:16:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-05-17T15:16:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_19 2024-05-17T15:16:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:16:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:16:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:16:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-05-17T15:16:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:16:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:16:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:16:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:16:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:16:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:16:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:16:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:16:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-05-17T15:16:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:16:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:16:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:16:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:16:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:16:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:16:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:16:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:16:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:16:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:16:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:16:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:16:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-05-17T15:16:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:16:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:16:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:16:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:16:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:16:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:16:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:16:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-05-17T15:16:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:16:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_19 2024-05-17T15:16:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:16:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:16:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2024-05-17T15:16:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:16:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_19 2024-05-17T15:16:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_19.status 2024-05-17T15:16:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_19.status 2024-05-17T15:16:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:16:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_19.status 2024-05-17T15:16:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_19.status 2024-05-17T15:16:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_19.status 2024-05-17T15:16:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:16:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_19.status 2024-05-17T15:16:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-05-17T15:16:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_19 2024-05-17T15:16:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-05-17T15:16:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 122 2024-05-17T15:16:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_19.status 2024-05-17T15:16:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_19 2024-05-17T15:16:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_19.status 2024-05-17T15:16:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-05-17T15:16:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:16:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=122 2024-05-17T15:16:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:16:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:16:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_19.status 2024-05-17T15:16:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_19 2024-05-17T15:16:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_19.status 2024-05-17T15:16:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-05-17T15:16:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:16:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:16:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:16:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:16:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:16:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 122 = '' 2024-05-17T15:16:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:16:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_19.status 2024-05-17T15:16:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_19 2024-05-17T15:16:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_19.status 2024-05-17T15:16:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-05-17T15:16:42.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:16:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:16:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:16:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:16:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:16:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 122 = '' 2024-05-17T15:16:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:16:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:16:42.823 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2024-05-17T15:16:42.999 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5526285500637657) > weight (0.25) 2024-05-17T15:16:42.999 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.37964353767002423) > weight (0.25) 2024-05-17T15:16:42.999 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9584758217725026) > weight (0.25) 2024-05-17T15:16:42.999 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-05-17T15:16:43.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-05-17T15:16:43.000 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-05-17T15:16:43.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi195.stderr:2024-05-17T15:16:42.999+0000 7fa8f4249640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 38072) UID: 1000 2024-05-17T15:16:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_19.status 2024-05-17T15:16:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_19 2024-05-17T15:16:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_19.status 2024-05-17T15:16:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-05-17T15:16:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:16:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:16:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=122 2024-05-17T15:16:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 122 = 122 2024-05-17T15:16:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-05-17T15:16:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_19 2024-05-17T15:16:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:16:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:16:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:16:44.070 INFO:tasks.ceph.cluster1.osd.2.smithi134.stderr:2024-05-17T15:16:44.063+0000 7f435fe18640 -1 reset not still connected to 0x55702102b5f0 2024-05-17T15:16:44.071 INFO:tasks.ceph.cluster1.osd.2.smithi134.stderr:2024-05-17T15:16:44.065+0000 7f435fe18640 -1 reset not still connected to 0x557022a368f0 2024-05-17T15:16:44.073 INFO:tasks.ceph.cluster1.osd.2.smithi134.stderr:2024-05-17T15:16:44.069+0000 7f435fe18640 -1 reset not still connected to 0x55702331a680 2024-05-17T15:16:44.073 INFO:tasks.ceph.cluster1.osd.2.smithi134.stderr:2024-05-17T15:16:44.069+0000 7f435fe18640 -1 reset not still connected to 0x5570255e9c70 2024-05-17T15:16:44.074 INFO:tasks.ceph.cluster1.osd.2.smithi134.stderr:2024-05-17T15:16:44.069+0000 7f435fe18640 -1 reset not still connected to 0x557025615c70 2024-05-17T15:16:44.074 INFO:tasks.ceph.cluster1.osd.2.smithi134.stderr:2024-05-17T15:16:44.069+0000 7f435fe18640 -1 reset not still connected to 0x5570259b3ad0 2024-05-17T15:16:44.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:16:44.130+0000 7f8ccd048640 -1 rbd::mirror::ImageReplayer: 0x55b0fc8b4c80 [3/bef15777-4cd7-47fb-8829-12dabc805b90] start: already running 2024-05-17T15:16:44.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:16:44.130+0000 7f8ccd048640 -1 rbd::mirror::ImageReplayer: 0x55b0fc8b4780 [3/c50f08e5-ef2e-4cc9-afd7-fb18dd3f03dc] start: already running 2024-05-17T15:16:44.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T15:16:44.137+0000 7f310ff9c640 -1 rbd::mirror::ImageReplayer: 0x558e702d5b80 [3/36a3913d-378a-412a-ad5b-9c269860173e] start: already running 2024-05-17T15:16:44.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T15:16:44.137+0000 7f310ff9c640 -1 rbd::mirror::ImageReplayer: 0x558e70e08a00 [3/46ae8559-1830-4c3d-adfc-c8f96345efad] start: already running 2024-05-17T15:16:44.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T15:16:44.137+0000 7f310ff9c640 -1 rbd::mirror::ImageReplayer: 0x558e702d5180 [3/5191746b-1578-4957-a477-41bde526d1e0] start: already running 2024-05-17T15:16:44.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T15:16:44.137+0000 7f310ff9c640 -1 rbd::mirror::ImageReplayer: 0x558e702d5900 [3/32af895d-8b5f-4d18-99db-b2ad45a7ac69] start: already running 2024-05-17T15:16:44.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:16:44.142+0000 7f8ccd048640 -1 rbd::mirror::ImageReplayer: 0x55b0fc8b5b80 [3/90f08d14-8ef0-4bff-9f0d-d804ed59aad1] start: already running 2024-05-17T15:16:44.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:16:44.142+0000 7f8ccd048640 -1 rbd::mirror::ImageReplayer: 0x55b0fc8b4a00 [3/b40480e1-3aef-46bb-a86f-05a1ee0d2d26] start: already running 2024-05-17T15:16:44.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:16:44.154+0000 7f8ccc847640 -1 rbd::mirror::ImageReplayer: 0x55b0fc8b5180 [3/ceaa7752-4708-4453-aad7-8c9fe4b5ad62] start: already running 2024-05-17T15:16:44.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:16:44.154+0000 7f8ccc847640 -1 rbd::mirror::ImageReplayer: 0x55b0fc8b5680 [3/5f4f313b-d9df-420f-8c92-2d9e481184bc] start: already running 2024-05-17T15:16:44.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:16:44.154+0000 7f8ccc847640 -1 rbd::mirror::ImageReplayer: 0x55b0fc8b4500 [3/7fede8a9-d664-48e9-b43a-740b49fc17be] start: already running 2024-05-17T15:16:44.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:16:44.154+0000 7f8ccc847640 -1 rbd::mirror::ImageReplayer: 0x55b0fd042000 [3/9e41e805-f938-4777-a0dd-ce32b1acfcf1] start: already running 2024-05-17T15:16:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-05-17T15:16:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_19': 2024-05-17T15:16:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:16:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:16:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:16:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 128479b7028b 2024-05-17T15:16:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.128479b7028b 2024-05-17T15:16:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:16:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:16:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:16:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:16:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:07:50 2024 2024-05-17T15:16:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:07:50 2024 2024-05-17T15:16:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:07:50 2024 2024-05-17T15:16:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:16:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:16:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:16:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 3df98478-a489-4a33-b01c-132fa8241534 2024-05-17T15:16:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:16:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_19 snap 2024-05-17T15:16:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_19 2024-05-17T15:16:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:16:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_19.export 2024-05-17T15:16:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_19.export 2024-05-17T15:16:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_19.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_19.export 2024-05-17T15:16:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_19.export 2024-05-17T15:16:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:16:44.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_19.export 2024-05-17T15:16:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:16:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_19.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_19.export 2024-05-17T15:16:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_19.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_19.export 2024-05-17T15:16:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:16:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:16:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_20 2024-05-17T15:16:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_20 snap 2024-05-17T15:16:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:16:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_20 2024-05-17T15:16:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:16:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:16:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-05-17T15:16:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_20 2024-05-17T15:16:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-05-17T15:16:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_20 2024-05-17T15:16:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:16:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:16:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:16:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-05-17T15:16:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:16:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:16:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_20 2024-05-17T15:16:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:16:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:16:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:16:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-05-17T15:16:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:16:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:16:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:16:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:16:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:16:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:16:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:16:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:16:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-05-17T15:16:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:16:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:16:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:16:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:16:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:16:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:16:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:16:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:16:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:16:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:16:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:16:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:16:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-05-17T15:16:48.000 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-05-17T15:16:48.001 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:16:48.001 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-05-17T15:16:48.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-05-17T15:16:48.001 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-05-17T15:16:48.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-05-17T15:16:48.004 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2024-05-17T15:16:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:16:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-05-17T15:16:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:16:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:16:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:16:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:16:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:16:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:16:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:16:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:16:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:16:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:16:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:16:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:16:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-05-17T15:16:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:16:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2024-05-17T15:16:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:16:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:16:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:16:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:16:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:16:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:16:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:16:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:16:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:16:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:16:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:16:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:16:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:16:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:16:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:16:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:16:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2024-05-17T15:16:48.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2024-05-17T15:16:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:16:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2024-05-17T15:16:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:16:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:16:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:16:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:16:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:16:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:16:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:16:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:16:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:16:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:16:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:16:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:16:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:16:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:16:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:16:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:16:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2024-05-17T15:16:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:16:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 1 2024-05-17T15:16:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:16:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:16:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-05-17T15:16:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:16:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:16:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:16:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:16:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:16:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:16:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:16:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:16:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:16:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:16:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:16:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:16:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-05-17T15:16:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:16:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:49.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:49.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-05-17T15:16:49.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:16:49.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:49.465 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:16:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:16:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:16:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:16:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:16:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:16:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-05-17T15:16:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:16:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2024-05-17T15:16:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:16:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:16:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:16:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:16:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:16:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:16:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:16:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:16:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:16:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:16:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:16:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:16:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:16:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:16:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:16:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:16:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2024-05-17T15:16:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:16:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2024-05-17T15:16:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:16:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:16:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:16:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:16:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:16:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:16:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:16:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:16:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:16:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:16:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:16:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:16:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:16:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:16:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:16:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:16:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:16:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:16:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:16:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2024-05-17T15:16:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:16:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 2 2024-05-17T15:16:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:16:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:16:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-05-17T15:16:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:16:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:16:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:16:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:16:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:16:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:16:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:16:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:16:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:16:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:16:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:16:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:16:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-05-17T15:16:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:16:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-05-17T15:16:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:16:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:16:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:16:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:16:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:16:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:16:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:16:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:16:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:16:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:16:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:16:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:16:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-05-17T15:16:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:16:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2024-05-17T15:16:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:16:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:16:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:16:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:16:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:16:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:16:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:16:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:16:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:16:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:16:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:16:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:16:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:16:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:16:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:16:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:16:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2024-05-17T15:16:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:16:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:52.109 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:52.109 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2024-05-17T15:16:52.109 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:16:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:16:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:16:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:16:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:16:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:16:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:16:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:16:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:16:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:16:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:16:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:16:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:16:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:16:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:16:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:16:52.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:16:52.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:16:52.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:16:52.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2024-05-17T15:16:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:16:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 4 2024-05-17T15:16:53.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~222s 2024-05-17T15:16:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:16:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:16:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-05-17T15:16:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:16:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:16:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:16:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:16:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:16:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:16:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:16:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:16:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:16:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:16:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:16:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:16:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:16:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:16:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:16:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-05-17T15:16:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:16:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-05-17T15:16:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:16:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:16:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:16:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:16:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:16:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:16:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:16:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:16:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:16:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:16:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:16:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:16:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:16:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:16:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:16:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-05-17T15:16:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:16:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2024-05-17T15:16:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:16:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:16:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:16:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:16:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:16:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:16:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:16:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:16:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:16:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:16:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:16:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:16:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:16:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:16:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:16:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:16:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:16:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2024-05-17T15:16:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:16:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:16:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2024-05-17T15:16:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:16:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:16:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:16:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:16:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:16:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:16:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:16:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:16:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:16:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:16:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:16:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:16:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:16:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:16:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:16:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:16:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:16:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:16:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:16:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:16:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:16:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:16:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:16:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:16:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:16:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:16:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:16:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:16:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:16:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:16:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:16:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:16:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2024-05-17T15:16:56.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:16:56.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:16:56.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:16:56.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:16:56.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 8 2024-05-17T15:16:59.034 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2024-05-17T15:17:04.439 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2024-05-17T15:17:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:17:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:17:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-05-17T15:17:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:17:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:17:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:17:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:17:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:17:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:17:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:17:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:17:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:17:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:17:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:17:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:17:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:17:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:17:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:17:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:17:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-05-17T15:17:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:17:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:17:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:17:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-05-17T15:17:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:17:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:17:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:17:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:17:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:17:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:17:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:17:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:17:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:17:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:17:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:17:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:17:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:17:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:17:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:17:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:17:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-05-17T15:17:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:17:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:17:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:17:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2024-05-17T15:17:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:17:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:17:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:17:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:17:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:17:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:17:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:17:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:17:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:17:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:17:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:17:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:17:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:17:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:17:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:17:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:17:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:17:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:17:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:17:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2024-05-17T15:17:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:17:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:17:04.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:17:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2024-05-17T15:17:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:17:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:17:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:17:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:17:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:17:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:17:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:17:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:17:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:17:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:17:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:17:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:17:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:17:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:17:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:17:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:17:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:17:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:17:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:17:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2024-05-17T15:17:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:17:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:17:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:17:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:17:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 8 2024-05-17T15:17:09.844 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~238s 2024-05-17T15:17:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:17:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:17:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-05-17T15:17:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:17:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:17:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:17:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:17:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:17:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:17:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:17:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:17:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:17:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:17:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:17:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:17:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:17:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:17:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:17:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:17:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-05-17T15:17:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:17:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:17:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:17:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-05-17T15:17:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:17:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:17:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:17:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:17:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:17:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:17:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:17:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:17:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:17:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:17:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:17:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:17:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:17:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:17:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:17:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:17:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-05-17T15:17:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:17:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:17:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:17:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2024-05-17T15:17:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:17:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:17:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:17:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:17:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:17:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:17:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:17:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:17:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:17:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:17:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:17:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:17:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:17:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:17:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:17:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:17:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:17:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:17:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:17:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2024-05-17T15:17:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:17:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:17:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:17:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2024-05-17T15:17:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:17:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:17:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:17:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:17:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:17:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:17:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:17:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:17:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:17:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:17:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:17:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:17:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:17:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:17:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:17:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:17:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:17:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:17:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:17:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2024-05-17T15:17:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:17:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:17:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:17:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:17:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 8 2024-05-17T15:17:15.248 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2024-05-17T15:17:20.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~249s 2024-05-17T15:17:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:17:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:17:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-05-17T15:17:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:17:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:17:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:17:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:17:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:17:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:17:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:17:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:17:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:17:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:17:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:17:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:17:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:17:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:17:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:17:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:17:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-05-17T15:17:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:17:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:17:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:17:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-05-17T15:17:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:17:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:17:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:17:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:17:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:17:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:17:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:17:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:17:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:17:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:17:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:17:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:17:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:17:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:17:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:17:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:17:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-05-17T15:17:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-05-17T15:17:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:17:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:17:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2024-05-17T15:17:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:17:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:17:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:17:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:17:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:17:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:17:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:17:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:17:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:17:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:17:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:17:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:17:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:17:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:17:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:17:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:17:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:17:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:17:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:17:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2024-05-17T15:17:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:17:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:17:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:17:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:17:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:17:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:17:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:17:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-05-17T15:17:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:17:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_20 2024-05-17T15:17:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:17:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:17:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2024-05-17T15:17:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:17:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_20 2024-05-17T15:17:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_20.status 2024-05-17T15:17:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_20.status 2024-05-17T15:17:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:17:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_20.status 2024-05-17T15:17:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_20.status 2024-05-17T15:17:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_20.status 2024-05-17T15:17:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:17:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_20.status 2024-05-17T15:17:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-05-17T15:17:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_20 2024-05-17T15:17:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-05-17T15:17:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 124 2024-05-17T15:17:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_20.status 2024-05-17T15:17:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_20 2024-05-17T15:17:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_20.status 2024-05-17T15:17:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-05-17T15:17:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:17:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=124 2024-05-17T15:17:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:17:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:17:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_20.status 2024-05-17T15:17:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_20 2024-05-17T15:17:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_20.status 2024-05-17T15:17:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-05-17T15:17:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:17:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:17:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:17:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:17:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:17:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 124 = '' 2024-05-17T15:17:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:17:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_20.status 2024-05-17T15:17:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_20 2024-05-17T15:17:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_20.status 2024-05-17T15:17:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-05-17T15:17:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:17:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:17:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:17:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:17:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:17:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 124 = '' 2024-05-17T15:17:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:17:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_20.status 2024-05-17T15:17:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_20 2024-05-17T15:17:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_20.status 2024-05-17T15:17:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-05-17T15:17:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:23.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=124 2024-05-17T15:17:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 124 = 124 2024-05-17T15:17:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-05-17T15:17:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:23.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_20 2024-05-17T15:17:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:17:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:17:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:17:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:17:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-05-17T15:17:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_20': 2024-05-17T15:17:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:17:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:17:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:17:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12939310b8e8 2024-05-17T15:17:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12939310b8e8 2024-05-17T15:17:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:17:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:17:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:17:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:17:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:08:01 2024 2024-05-17T15:17:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:08:01 2024 2024-05-17T15:17:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:08:01 2024 2024-05-17T15:17:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:17:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:17:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:17:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 99995afa-10de-4385-a379-455ec2be8a34 2024-05-17T15:17:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:17:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_20 snap 2024-05-17T15:17:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_20 2024-05-17T15:17:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:17:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:17:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_20.export 2024-05-17T15:17:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_20.export 2024-05-17T15:17:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_20.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_20.export 2024-05-17T15:17:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_20.export 2024-05-17T15:17:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:17:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_20.export 2024-05-17T15:17:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:17:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_20.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_20.export 2024-05-17T15:17:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_20.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_20.export 2024-05-17T15:17:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:17:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_21 2024-05-17T15:17:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_21 snap 2024-05-17T15:17:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_21 2024-05-17T15:17:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:17:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:17:26.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~254s 2024-05-17T15:17:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-05-17T15:17:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_21 2024-05-17T15:17:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-05-17T15:17:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_21 2024-05-17T15:17:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:17:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:17:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:17:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:17:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-05-17T15:17:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_21 2024-05-17T15:17:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:17:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:17:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:17:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-05-17T15:17:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:17:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:17:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:17:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:17:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:17:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:17:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:17:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:17:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:17:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-05-17T15:17:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:17:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:17:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:17:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:17:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:17:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:17:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:17:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:17:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:17:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:17:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:17:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:17:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:17:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:17:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:17:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:17:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-05-17T15:17:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-05-17T15:17:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:17:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:17:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2024-05-17T15:17:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:17:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:17:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:17:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:17:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:17:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:17:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:17:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:17:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:17:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:17:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:17:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:17:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:17:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:17:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:17:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:17:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2024-05-17T15:17:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:17:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:17:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:17:28.245 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:17:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:17:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:17:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:17:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-05-17T15:17:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:17:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_21 2024-05-17T15:17:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:17:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:17:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2024-05-17T15:17:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:17:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_21 2024-05-17T15:17:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_21.status 2024-05-17T15:17:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_21.status 2024-05-17T15:17:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:17:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_21.status 2024-05-17T15:17:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_21.status 2024-05-17T15:17:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_21.status 2024-05-17T15:17:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:17:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_21.status 2024-05-17T15:17:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-05-17T15:17:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_21 2024-05-17T15:17:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-05-17T15:17:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 126 2024-05-17T15:17:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_21.status 2024-05-17T15:17:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_21 2024-05-17T15:17:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_21.status 2024-05-17T15:17:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-05-17T15:17:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:17:29.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=126 2024-05-17T15:17:29.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:17:29.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:29.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:17:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_21.status 2024-05-17T15:17:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_21 2024-05-17T15:17:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_21.status 2024-05-17T15:17:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-05-17T15:17:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:17:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:17:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:17:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:17:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:17:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 126 = '' 2024-05-17T15:17:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:17:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_21.status 2024-05-17T15:17:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_21 2024-05-17T15:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_21.status 2024-05-17T15:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-05-17T15:17:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:17:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:17:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:17:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:17:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:17:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 126 = '' 2024-05-17T15:17:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:29.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:17:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_21.status 2024-05-17T15:17:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_21 2024-05-17T15:17:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_21.status 2024-05-17T15:17:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-05-17T15:17:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=126 2024-05-17T15:17:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 126 = 126 2024-05-17T15:17:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-05-17T15:17:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_21 2024-05-17T15:17:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:17:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:17:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:17:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:17:31.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~259s 2024-05-17T15:17:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-05-17T15:17:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_21': 2024-05-17T15:17:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:17:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:17:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:17:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d3ade11e6e 2024-05-17T15:17:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d3ade11e6e 2024-05-17T15:17:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:17:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:17:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:17:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:17:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:08:09 2024 2024-05-17T15:17:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:08:09 2024 2024-05-17T15:17:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:08:09 2024 2024-05-17T15:17:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:17:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:17:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:17:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 7fede8a9-d664-48e9-b43a-740b49fc17be 2024-05-17T15:17:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:17:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_21 snap 2024-05-17T15:17:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_21 2024-05-17T15:17:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:17:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:17:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_21.export 2024-05-17T15:17:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_21.export 2024-05-17T15:17:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_21.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_21.export 2024-05-17T15:17:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_21.export 2024-05-17T15:17:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:17:31.832 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_21.export 2024-05-17T15:17:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:17:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_21.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_21.export 2024-05-17T15:17:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_21.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_21.export 2024-05-17T15:17:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:17:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_22 2024-05-17T15:17:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_22 snap 2024-05-17T15:17:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:17:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:17:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:17:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-05-17T15:17:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:17:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-05-17T15:17:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:17:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:17:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:17:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:17:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:17:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-05-17T15:17:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:17:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:17:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:17:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:17:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-05-17T15:17:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:17:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:17:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:17:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:17:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:17:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:17:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:17:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:17:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:17:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-05-17T15:17:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:17:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:17:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:17:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:17:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:17:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:17:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:17:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:17:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:17:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:17:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:17:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:17:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:17:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:17:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:17:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:17:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-05-17T15:17:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-05-17T15:17:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:17:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:17:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-05-17T15:17:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:17:34.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:17:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:17:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:17:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:17:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:17:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:17:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:17:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:17:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:17:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:17:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:17:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:17:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:17:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:17:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:17:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-05-17T15:17:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-05-17T15:17:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:17:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:17:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2024-05-17T15:17:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:17:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:17:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:17:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:17:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:17:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:17:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:17:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:17:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:17:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:17:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:17:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:17:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:17:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:17:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:17:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:17:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:17:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:17:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:17:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2024-05-17T15:17:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-05-17T15:17:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:17:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:17:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_22 2024-05-17T15:17:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:17:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:17:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:17:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:17:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:17:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:17:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:17:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:17:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:17:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:17:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:17:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:17:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:17:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:17:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:17:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:17:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:17:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:17:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:17:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_22 2024-05-17T15:17:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:17:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:17:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:17:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:17:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:17:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:17:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:17:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-05-17T15:17:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:17:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:17:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:17:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:17:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2024-05-17T15:17:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:17:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:17:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_22.status 2024-05-17T15:17:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_22.status 2024-05-17T15:17:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:17:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_22.status 2024-05-17T15:17:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_22.status 2024-05-17T15:17:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_22.status 2024-05-17T15:17:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:17:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_22.status 2024-05-17T15:17:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-05-17T15:17:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:17:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-05-17T15:17:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 128 2024-05-17T15:17:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_22.status 2024-05-17T15:17:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:17:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_22.status 2024-05-17T15:17:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-05-17T15:17:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:17:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=128 2024-05-17T15:17:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:17:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:17:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_22.status 2024-05-17T15:17:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:17:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_22.status 2024-05-17T15:17:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-05-17T15:17:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=56 2024-05-17T15:17:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 128 = 56 2024-05-17T15:17:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:17:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_22.status 2024-05-17T15:17:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:17:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_22.status 2024-05-17T15:17:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-05-17T15:17:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=56 2024-05-17T15:17:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 128 = 56 2024-05-17T15:17:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:17:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_22.status 2024-05-17T15:17:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:17:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_22.status 2024-05-17T15:17:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-05-17T15:17:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:36.864 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~265s 2024-05-17T15:17:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:17:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:17:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:17:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:17:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:17:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 128 = '' 2024-05-17T15:17:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:17:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_22.status 2024-05-17T15:17:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:17:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_22.status 2024-05-17T15:17:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-05-17T15:17:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=128 2024-05-17T15:17:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 128 = 128 2024-05-17T15:17:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-05-17T15:17:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:17:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:17:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:17:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:17:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:17:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-05-17T15:17:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_22': 2024-05-17T15:17:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:17:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:17:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:17:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 113594126061 2024-05-17T15:17:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.113594126061 2024-05-17T15:17:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:17:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:17:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:17:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:17:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:08:23 2024 2024-05-17T15:17:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:08:23 2024 2024-05-17T15:17:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:08:23 2024 2024-05-17T15:17:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:17:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:17:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:17:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 7b475e11-a35f-4917-9f29-282df00e1098 2024-05-17T15:17:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:17:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_22 snap 2024-05-17T15:17:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_22 2024-05-17T15:17:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:17:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:17:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_22.export 2024-05-17T15:17:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_22.export 2024-05-17T15:17:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_22.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_22.export 2024-05-17T15:17:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_22.export 2024-05-17T15:17:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:17:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_22.export 2024-05-17T15:17:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:17:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_22.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_22.export 2024-05-17T15:17:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_22.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_22.export 2024-05-17T15:17:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:17:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_23 2024-05-17T15:17:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_23 snap 2024-05-17T15:17:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:17:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:17:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:17:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-05-17T15:17:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:17:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-05-17T15:17:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:17:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:17:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:17:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:17:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:17:42.269 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~270s 2024-05-17T15:17:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-05-17T15:17:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:17:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:17:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:17:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:17:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-05-17T15:17:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:17:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:17:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:17:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:17:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:17:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:17:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:17:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:17:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:17:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-05-17T15:17:42.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:17:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:17:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:17:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:17:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:17:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:17:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:17:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:17:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:17:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:17:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:17:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:17:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:17:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:17:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:17:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:17:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-05-17T15:17:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:17:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:17:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:17:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:17:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:17:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:17:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:17:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-05-17T15:17:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:17:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:17:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:17:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:17:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2024-05-17T15:17:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:17:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:17:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_23.status 2024-05-17T15:17:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_23.status 2024-05-17T15:17:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:17:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_23.status 2024-05-17T15:17:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_23.status 2024-05-17T15:17:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_23.status 2024-05-17T15:17:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:17:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_23.status 2024-05-17T15:17:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-05-17T15:17:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:17:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-05-17T15:17:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 130 2024-05-17T15:17:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_23.status 2024-05-17T15:17:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:17:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_23.status 2024-05-17T15:17:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-05-17T15:17:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:43.358 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:17:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=130 2024-05-17T15:17:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:17:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:43.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:17:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_23.status 2024-05-17T15:17:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:17:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_23.status 2024-05-17T15:17:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-05-17T15:17:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=57 2024-05-17T15:17:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 130 = 57 2024-05-17T15:17:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:17:44.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_23.status 2024-05-17T15:17:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:17:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_23.status 2024-05-17T15:17:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-05-17T15:17:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=57 2024-05-17T15:17:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 130 = 57 2024-05-17T15:17:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:17:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_23.status 2024-05-17T15:17:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:17:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_23.status 2024-05-17T15:17:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-05-17T15:17:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:17:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:17:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:17:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:17:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:17:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 130 = '' 2024-05-17T15:17:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:17:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_23.status 2024-05-17T15:17:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:17:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_23.status 2024-05-17T15:17:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-05-17T15:17:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=130 2024-05-17T15:17:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 130 = 130 2024-05-17T15:17:46.572 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-05-17T15:17:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:17:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:17:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:17:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:17:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:17:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-05-17T15:17:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_23': 2024-05-17T15:17:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:17:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:17:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:17:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1284c16da516 2024-05-17T15:17:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1284c16da516 2024-05-17T15:17:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:17:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:17:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:17:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:17:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:08:33 2024 2024-05-17T15:17:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:08:33 2024 2024-05-17T15:17:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:08:33 2024 2024-05-17T15:17:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:17:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:17:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:17:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 2968dbed-dc66-4e82-be5f-bc6e3322a6c1 2024-05-17T15:17:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:17:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_23 snap 2024-05-17T15:17:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_23 2024-05-17T15:17:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:17:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:17:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_23.export 2024-05-17T15:17:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_23.export 2024-05-17T15:17:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_23.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_23.export 2024-05-17T15:17:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_23.export 2024-05-17T15:17:47.673 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~276s 2024-05-17T15:17:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:17:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_23.export 2024-05-17T15:17:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:17:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_23.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_23.export 2024-05-17T15:17:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_23.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_23.export 2024-05-17T15:17:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:17:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_24 2024-05-17T15:17:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_24 snap 2024-05-17T15:17:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:17:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:17:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:17:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-05-17T15:17:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:17:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-05-17T15:17:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:17:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:17:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:17:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:17:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:17:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-05-17T15:17:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:17:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:17:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:17:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:17:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-05-17T15:17:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:17:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:17:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:17:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:17:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:17:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:17:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:17:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:17:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:17:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-05-17T15:17:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:17:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:17:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:17:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:17:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:17:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:17:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:17:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:17:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:17:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:17:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:17:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:17:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:17:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:17:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:17:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:17:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-05-17T15:17:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-05-17T15:17:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:17:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:17:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-05-17T15:17:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:17:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:17:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:17:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:17:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:17:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:17:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:17:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:17:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:17:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:17:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:17:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:17:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:17:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:17:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:17:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:17:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-05-17T15:17:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-05-17T15:17:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:17:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:17:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2024-05-17T15:17:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:17:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:17:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:17:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:17:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:17:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:17:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:17:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:17:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:17:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:17:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:17:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:17:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:17:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:17:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:17:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:17:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:17:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:17:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:17:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2024-05-17T15:17:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:17:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:17:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:17:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:17:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:17:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:17:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:17:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-05-17T15:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2024-05-17T15:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_24.status 2024-05-17T15:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_24.status 2024-05-17T15:17:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:17:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_24.status 2024-05-17T15:17:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_24.status 2024-05-17T15:17:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_24.status 2024-05-17T15:17:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:17:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_24.status 2024-05-17T15:17:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-05-17T15:17:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:17:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-05-17T15:17:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 132 2024-05-17T15:17:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_24.status 2024-05-17T15:17:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:17:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_24.status 2024-05-17T15:17:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-05-17T15:17:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:17:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=132 2024-05-17T15:17:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:17:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_24.status 2024-05-17T15:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_24.status 2024-05-17T15:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-05-17T15:17:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=58 2024-05-17T15:17:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 132 = 58 2024-05-17T15:17:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:17:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_24.status 2024-05-17T15:17:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:17:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_24.status 2024-05-17T15:17:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-05-17T15:17:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:17:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:17:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:17:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:17:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:17:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 132 = '' 2024-05-17T15:17:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:17:53.077 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~281s 2024-05-17T15:17:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_24.status 2024-05-17T15:17:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:17:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_24.status 2024-05-17T15:17:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-05-17T15:17:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:17:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:17:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:17:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:17:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:17:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 132 = '' 2024-05-17T15:17:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:17:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_24.status 2024-05-17T15:17:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:17:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_24.status 2024-05-17T15:17:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-05-17T15:17:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:17:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=132 2024-05-17T15:17:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 132 = 132 2024-05-17T15:17:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-05-17T15:17:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:17:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:17:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:17:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:17:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:17:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-05-17T15:17:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_24': 2024-05-17T15:17:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:17:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:17:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:17:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12a3ab3a6e1a 2024-05-17T15:17:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12a3ab3a6e1a 2024-05-17T15:17:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:17:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:17:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:17:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:17:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:08:48 2024 2024-05-17T15:17:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:08:48 2024 2024-05-17T15:17:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:08:48 2024 2024-05-17T15:17:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:17:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:17:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:17:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: d95d4f33-deca-446d-88ca-2fd4ef56f554 2024-05-17T15:17:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:17:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_24 snap 2024-05-17T15:17:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_24 2024-05-17T15:17:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:17:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:17:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_24.export 2024-05-17T15:17:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_24.export 2024-05-17T15:17:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_24.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_24.export 2024-05-17T15:17:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_24.export 2024-05-17T15:17:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:17:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_24.export 2024-05-17T15:17:56.452 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:17:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_24.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_24.export 2024-05-17T15:17:56.512 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_24.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_24.export 2024-05-17T15:17:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:17:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_25 2024-05-17T15:17:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_25 snap 2024-05-17T15:17:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:17:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:17:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:17:57.009 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6549348877603127) > weight (0.25) 2024-05-17T15:17:57.009 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-05-17T15:17:57.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-05-17T15:17:57.010 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-05-17T15:17:57.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi195.stderr:2024-05-17T15:17:57.009+0000 7f9d5e8ed640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 44557) UID: 1000 2024-05-17T15:17:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-05-17T15:17:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:17:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-05-17T15:17:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:17:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:17:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:17:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:17:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:17:58.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~286s 2024-05-17T15:17:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-05-17T15:17:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:17:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:17:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:17:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:17:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-05-17T15:17:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:17:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:17:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:17:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:17:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:17:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:17:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:17:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:17:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:17:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-05-17T15:17:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:17:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:17:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:17:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:17:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:17:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:17:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:17:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:17:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:17:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:17:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:17:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:17:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:17:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:17:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:17:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:17:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:17:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-05-17T15:17:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-05-17T15:17:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:17:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:17:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:17:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2024-05-17T15:17:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:17:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:17:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:17:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:17:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:17:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:17:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:17:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:17:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:17:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:17:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:17:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:17:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:17:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:17:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:17:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:17:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:17:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:17:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:17:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:17:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:17:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:17:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:17:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:17:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:17:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:17:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:17:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:17:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:17:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:17:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2024-05-17T15:17:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:17:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:17:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:17:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:17:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:17:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:17:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:17:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:17:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-05-17T15:17:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:17:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:17:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:17:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:17:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2024-05-17T15:17:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:17:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:17:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_25.status 2024-05-17T15:17:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_25.status 2024-05-17T15:17:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:17:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_25.status 2024-05-17T15:17:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_25.status 2024-05-17T15:17:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_25.status 2024-05-17T15:17:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:17:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_25.status 2024-05-17T15:17:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-05-17T15:17:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:17:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-05-17T15:17:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 134 2024-05-17T15:17:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_25.status 2024-05-17T15:17:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:17:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:17:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_25.status 2024-05-17T15:17:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-05-17T15:17:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:17:59.792 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:17:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=134 2024-05-17T15:17:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:17:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:17:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:17:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_25.status 2024-05-17T15:17:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:17:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:17:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:17:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_25.status 2024-05-17T15:17:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-05-17T15:17:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:18:00.005 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7529144626970344) > weight (0.25) 2024-05-17T15:18:00.005 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8672195424872865) > weight (0.25) 2024-05-17T15:18:00.005 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.34333223699697446) > weight (0.25) 2024-05-17T15:18:00.005 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5442398063846591) > weight (0.25) 2024-05-17T15:18:00.005 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2024-05-17T15:18:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:18:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=59 2024-05-17T15:18:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 134 = 59 2024-05-17T15:18:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:18:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:18:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_25.status 2024-05-17T15:18:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:18:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:18:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_25.status 2024-05-17T15:18:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-05-17T15:18:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:18:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:18:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:18:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:18:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:18:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:18:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:18:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 134 = '' 2024-05-17T15:18:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:18:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:18:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_25.status 2024-05-17T15:18:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:18:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:18:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_25.status 2024-05-17T15:18:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-05-17T15:18:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:18:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:18:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:18:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:18:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:18:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:18:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:18:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 134 = '' 2024-05-17T15:18:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:18:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:18:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_25.status 2024-05-17T15:18:02.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:18:02.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:02.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:18:02.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_25.status 2024-05-17T15:18:02.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-05-17T15:18:02.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:18:03.011 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-05-17T15:18:03.011 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:18:03.011 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-05-17T15:18:03.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-05-17T15:18:03.012 DEBUG:teuthology.orchestra.run.smithi195:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-05-17T15:18:03.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-05-17T15:18:03.014 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2024-05-17T15:18:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:18:03.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=134 2024-05-17T15:18:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 134 = 134 2024-05-17T15:18:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:18:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-05-17T15:18:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:18:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:18:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:18:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:18:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:18:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:18:03.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~292s 2024-05-17T15:18:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-05-17T15:18:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_25': 2024-05-17T15:18:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:18:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:18:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:18:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12a3ba5aa645 2024-05-17T15:18:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12a3ba5aa645 2024-05-17T15:18:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:18:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:18:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:18:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:18:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:08:59 2024 2024-05-17T15:18:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:08:59 2024 2024-05-17T15:18:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:08:59 2024 2024-05-17T15:18:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:18:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:18:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:18:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 5f4f313b-d9df-420f-8c92-2d9e481184bc 2024-05-17T15:18:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:18:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:18:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_25 snap 2024-05-17T15:18:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_25 2024-05-17T15:18:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:18:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:18:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_25.export 2024-05-17T15:18:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_25.export 2024-05-17T15:18:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_25.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_25.export 2024-05-17T15:18:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_25.export 2024-05-17T15:18:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:18:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_25.export 2024-05-17T15:18:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:18:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_25.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_25.export 2024-05-17T15:18:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_25.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_25.export 2024-05-17T15:18:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:18:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:18:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_26 2024-05-17T15:18:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_26 snap 2024-05-17T15:18:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:18:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:18:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:18:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:18:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-05-17T15:18:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:18:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:18:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-05-17T15:18:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:18:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:18:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:18:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:18:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:18:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:18:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-05-17T15:18:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:18:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:18:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:18:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:18:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:18:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-05-17T15:18:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:18:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:18:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:18:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:18:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:18:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:18:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:18:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:18:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:18:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:18:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:18:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-05-17T15:18:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:18:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:18:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:18:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:18:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:18:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:18:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:18:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:18:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:18:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:18:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:18:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:18:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:18:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:18:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:18:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:18:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:18:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:18:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:18:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:18:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-05-17T15:18:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-05-17T15:18:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:18:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:18:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:18:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-05-17T15:18:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:18:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:18:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:18:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:18:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:18:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:18:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:18:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:18:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:18:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:18:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:18:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:18:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:18:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:18:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:18:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:18:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:18:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:18:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:18:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:18:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:18:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:18:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:18:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:18:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:18:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:18:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:18:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:18:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:18:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:18:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:18:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:18:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:18:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-05-17T15:18:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-05-17T15:18:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:18:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:18:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:18:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2024-05-17T15:18:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:18:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:18:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:18:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:18:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:18:06.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2024-05-17T15:18:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-05-17T15:18:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:18:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:18:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:18:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_26 2024-05-17T15:18:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:18:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:18:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:18:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:18:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:18:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:18:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:18:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:18:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:18:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:18:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:18:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:18:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:18:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:18:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:18:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:18:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:18:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:18:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:18:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:18:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:18:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:18:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:18:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:18:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:18:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:18:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:18:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:18:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:18:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:18:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:18:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:18:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:18:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_26 2024-05-17T15:18:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:18:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:18:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:18:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:18:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:18:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:18:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:18:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:18:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-05-17T15:18:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:18:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:18:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:18:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:18:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:18:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2024-05-17T15:18:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:18:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:18:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:18:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_26.status 2024-05-17T15:18:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_26.status 2024-05-17T15:18:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:18:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_26.status 2024-05-17T15:18:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_26.status 2024-05-17T15:18:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_26.status 2024-05-17T15:18:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:18:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_26.status 2024-05-17T15:18:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-05-17T15:18:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:18:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:18:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-05-17T15:18:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Snapshot ID: 136 2024-05-17T15:18:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.0i1dljubcH/cluster2-mirror-image_26.status 2024-05-17T15:18:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:18:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:18:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_26.status 2024-05-17T15:18:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-05-17T15:18:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:18:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-05-17T15:18:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snapshot_id=136 2024-05-17T15:18:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:18:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:18:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.2 2024-05-17T15:18:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_26.status 2024-05-17T15:18:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:18:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:18:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_26.status 2024-05-17T15:18:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-05-17T15:18:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:18:08.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:18:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=60 2024-05-17T15:18:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 136 = 60 2024-05-17T15:18:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:18:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.4 2024-05-17T15:18:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_26.status 2024-05-17T15:18:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:18:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:18:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_26.status 2024-05-17T15:18:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-05-17T15:18:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:18:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:18:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:18:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:18:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:18:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:18:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:18:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 136 = '' 2024-05-17T15:18:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:18:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 0.8 2024-05-17T15:18:09.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~297s 2024-05-17T15:18:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_26.status 2024-05-17T15:18:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:18:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:18:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_26.status 2024-05-17T15:18:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-05-17T15:18:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:18:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ true 2024-05-17T15:18:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:18:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:-:1.1: Document is empty 2024-05-17T15:18:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: 2024-05-17T15:18:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:^ 2024-05-17T15:18:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id= 2024-05-17T15:18:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 136 = '' 2024-05-17T15:18:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-05-17T15:18:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1.6 2024-05-17T15:18:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.0i1dljubcH/cluster1-mirror-image_26.status 2024-05-17T15:18:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:18:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:18:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_26.status 2024-05-17T15:18:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-05-17T15:18:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-05-17T15:18:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-05-17T15:18:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local primary_snapshot_id=136 2024-05-17T15:18:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test 136 = 136 2024-05-17T15:18:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:18:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-05-17T15:18:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:18:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:18:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:18:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:18:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-05-17T15:18:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:18:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-05-17T15:18:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_26': 2024-05-17T15:18:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:18:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:18:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:18:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 11356b06411f 2024-05-17T15:18:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.11356b06411f 2024-05-17T15:18:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:18:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:18:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:18:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:18:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:09:13 2024 2024-05-17T15:18:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:09:13 2024 2024-05-17T15:18:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:09:13 2024 2024-05-17T15:18:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: protected: False 2024-05-17T15:18:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:18:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:18:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 9d50b403-8aa5-4da6-8e72-4a3f61c9358b 2024-05-17T15:18:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:18:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:18:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ compare_image_snaps mirror image_26 snap 2024-05-17T15:18:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_26 2024-05-17T15:18:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:18:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret=0 2024-05-17T15:18:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local rmt_export=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_26.export 2024-05-17T15:18:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local loc_export=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_26.export 2024-05-17T15:18:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_26.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_26.export 2024-05-17T15:18:12.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.0i1dljubcH/cluster2-mirror-image_26.export 2024-05-17T15:18:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:18:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.0i1dljubcH/cluster1-mirror-image_26.export 2024-05-17T15:18:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-05-17T15:18:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cmp /tmp/tmp.0i1dljubcH/cluster2-mirror-image_26.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_26.export 2024-05-17T15:18:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -f /tmp/tmp.0i1dljubcH/cluster2-mirror-image_26.export /tmp/tmp.0i1dljubcH/cluster1-mirror-image_26.export 2024-05-17T15:18:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:18:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-05-17T15:18:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ image=image_27 2024-05-17T15:18:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ create_snap cluster2 mirror image_27 snap 2024-05-17T15:18:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:18:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_27 2024-05-17T15:18:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local snap_name=snap 2024-05-17T15:18:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-05-17T15:18:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-05-17T15:18:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:18:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_27 2024-05-17T15:18:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-05-17T15:18:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:18:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_27 2024-05-17T15:18:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local state=started 2024-05-17T15:18:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local s 2024-05-17T15:18:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:18:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ sleep 1 2024-05-17T15:18:14.695 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~303s 2024-05-17T15:18:14.695 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-05-17T15:18:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-05-17T15:18:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T15:18:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_27 2024-05-17T15:18:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local test_state=started 2024-05-17T15:18:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_result 2024-05-17T15:18:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local current_state=stopped 2024-05-17T15:18:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-05-17T15:18:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_instance=cluster1 2024-05-17T15:18:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:18:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ grep -i state 2024-05-17T15:18:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1 2024-05-17T15:18:14.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance=cluster1 2024-05-17T15:18:14.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local loop_instance 2024-05-17T15:18:14.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-05-17T15:18:14.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sleep 0 2024-05-17T15:18:14.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-05-17T15:18:14.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ seq 0 3 2024-05-17T15:18:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:18:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-05-17T15:18:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T15:18:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:18:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:18:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T15:18:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T15:18:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:18:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:18:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:18:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T15:18:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T15:18:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:18:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:18:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T15:18:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:18:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-05-17T15:18:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:0 2024-05-17T15:18:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:18:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:18:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-05-17T15:18:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:0 2024-05-17T15:18:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:18:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:18:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:18:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T15:18:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 0 = cluster1:0 ']' 2024-05-17T15:18:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:18:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:18:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=0 2024-05-17T15:18:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=0 2024-05-17T15:18:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-05-17T15:18:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:18:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:18:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-05-17T15:18:14.799 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-05-17T15:18:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-05-17T15:18:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:18:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:18:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:18:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-05-17T15:18:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T15:18:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:18:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:18:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T15:18:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T15:18:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:18:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:18:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:18:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T15:18:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T15:18:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:18:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:18:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T15:18:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:18:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-05-17T15:18:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:1 2024-05-17T15:18:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:18:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:18:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-05-17T15:18:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:1 2024-05-17T15:18:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:18:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:18:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:18:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T15:18:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 1 = cluster1:1 ']' 2024-05-17T15:18:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:18:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:18:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=1 2024-05-17T15:18:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=1 2024-05-17T15:18:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-05-17T15:18:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:18:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:18:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-05-17T15:18:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-05-17T15:18:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-05-17T15:18:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:18:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:18:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:18:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2024-05-17T15:18:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T15:18:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:18:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:18:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T15:18:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T15:18:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:18:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:18:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:18:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T15:18:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T15:18:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:18:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:18:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T15:18:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:18:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-05-17T15:18:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:2 2024-05-17T15:18:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:18:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:18:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-05-17T15:18:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:2 2024-05-17T15:18:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:18:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:18:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:18:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T15:18:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 2 = cluster1:2 ']' 2024-05-17T15:18:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:18:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:18:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=2 2024-05-17T15:18:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=2 2024-05-17T15:18:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-05-17T15:18:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:18:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:18:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-05-17T15:18:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2024-05-17T15:18:15.100 INFO:tasks.ceph.cluster2.osd.1:Sent signal 15 2024-05-17T15:18:15.100 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2024-05-17T15:18:15.101 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2024-05-17T15:18:15.101 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2024-05-17T15:18:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-05-17T15:18:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd mirror status : get status for rbd mirror 2024-05-17T15:18:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:admin_socket: invalid command 2024-05-17T15:18:15.102 INFO:tasks.ceph.cluster2.osd.1.smithi195.stderr:2024-05-17T15:18:15.099+0000 7f83a982a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 (PID: 33546) UID: 0 2024-05-17T15:18:15.102 INFO:tasks.ceph.cluster2.osd.1.smithi195.stderr:2024-05-17T15:18:15.099+0000 7f83a982a640 -1 osd.1 244 *** Got signal Terminated *** 2024-05-17T15:18:15.102 INFO:tasks.ceph.cluster2.osd.1.smithi195.stderr:2024-05-17T15:18:15.099+0000 7f83a982a640 -1 osd.1 244 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-17T15:18:15.103 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:18:15.099+0000 7f7a5b1b7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 33548) UID: 0 2024-05-17T15:18:15.103 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:18:15.099+0000 7f7a5b1b7640 -1 osd.2 244 *** Got signal Terminated *** 2024-05-17T15:18:15.103 INFO:tasks.ceph.cluster2.osd.2.smithi195.stderr:2024-05-17T15:18:15.099+0000 7f7a5b1b7640 -1 osd.2 244 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-17T15:18:15.103 INFO:tasks.ceph.cluster2.mon.a.smithi195.stderr:2024-05-17T15:18:15.100+0000 7f4d6f61c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 33238) UID: 0 2024-05-17T15:18:15.103 INFO:tasks.ceph.cluster2.mon.a.smithi195.stderr:2024-05-17T15:18:15.100+0000 7f4d6f61c640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-17T15:18:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T15:18:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2024-05-17T15:18:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T15:18:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ shift 2024-05-17T15:18:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T15:18:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T15:18:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T15:18:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T15:18:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T15:18:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T15:18:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T15:18:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T15:18:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T15:18:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:18:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T15:18:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:18:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-05-17T15:18:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local local_cluster=cluster1:3 2024-05-17T15:18:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster=cluster1 2024-05-17T15:18:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance 2024-05-17T15:18:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-05-17T15:18:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local val=cluster1:3 2024-05-17T15:18:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local cluster_var_name=local_cluster 2024-05-17T15:18:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ local instance_var_name=instance 2024-05-17T15:18:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T15:18:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T15:18:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ '[' 3 = cluster1:3 ']' 2024-05-17T15:18:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval local_cluster=cluster1 2024-05-17T15:18:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ local_cluster=cluster1 2024-05-17T15:18:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ eval instance=3 2024-05-17T15:18:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ instance=3 2024-05-17T15:18:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-05-17T15:18:15.110 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T15:18:15.100+0000 7fde58f87640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 37869) UID: 0 2024-05-17T15:18:15.110 INFO:tasks.ceph.cluster2.mgr.x.smithi134.stderr:2024-05-17T15:18:15.100+0000 7fde58f87640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-05-17T15:18:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:18:15.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:18:15.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-05-17T15:18:15.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2024-05-17T15:18:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ return 0 2024-05-17T15:18:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status_result=' "state": "Replaying"' 2024-05-17T15:18:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo ' "state": "Replaying"' 2024-05-17T15:18:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ grep -i Replaying 2024-05-17T15:18:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "state": "Replaying" 2024-05-17T15:18:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ current_state=started 2024-05-17T15:18:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test started = started 2024-05-17T15:18:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 0 2024-05-17T15:18:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-05-17T15:18:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:18:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:18:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_27 2024-05-17T15:18:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = journal ']' 2024-05-17T15:18:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' snapshot = snapshot ']' 2024-05-17T15:18:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2024-05-17T15:18:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_cluster=cluster1 2024-05-17T15:18:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:18:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_27 2024-05-17T15:18:15.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster2-mirror-image_27.status 2024-05-17T15:18:15.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster2-mirror-image_27.status 2024-05-17T15:18:15.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:18:15.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local status_log=/tmp/tmp.0i1dljubcH/cluster2-mirror-image_27.status 2024-05-17T15:18:15.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ mkfname cluster1-mirror-image_27.status 2024-05-17T15:18:15.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo cluster1-mirror-image_27.status 2024-05-17T15:18:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ sed -e 's|[/ ]|_|g' 2024-05-17T15:18:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local local_status_log=/tmp/tmp.0i1dljubcH/cluster1-mirror-image_27.status 2024-05-17T15:18:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-05-17T15:18:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T15:18:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pool=mirror 2024-05-17T15:18:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local image=image_27 2024-05-17T15:18:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-05-17T15:19:40.007 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-05-17T15:19:40.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-05-17T15:19:40.009 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-05-17T15:19:40.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T15:19:40.007+0000 7fc686478640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 51985) UID: 1000 2024-05-17T15:19:44.010 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-05-17T15:19:44.010 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:19:50.015 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-05-17T15:19:50.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-05-17T15:19:50.016 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-05-17T15:19:50.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr:2024-05-17T15:19:50.015+0000 7f15d9a11640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 56208) UID: 1000 2024-05-17T15:19:55.017 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-05-17T15:19:55.017 INFO:teuthology.orchestra.run:waiting for 600 2024-05-17T15:20:08.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T15:20:08.335+0000 7fc685c77640 -1 rbd::mirror::InstanceWatcher: 0x5557e5e4ae08 handle_break_instance_lock: error breaking instance lock: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:08.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T15:20:08.336+0000 7fc671c4f640 -1 rbd::mirror::InstanceWatcher: 0x5557e5e4ae08 handle_remove_instance_object: error removing rbd_mirror_instance.4743 object: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:08.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T15:20:08.336+0000 7fc685476640 -1 rbd::mirror::InstanceWatcher: 0x5557e5e4b188 handle_break_instance_lock: error breaking instance lock: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:08.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T15:20:08.337+0000 7fc67144e640 -1 rbd::mirror::InstanceWatcher: 0x5557e5e4b188 handle_remove_instance_object: error removing rbd_mirror_instance.4989 object: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:08.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T15:20:08.338+0000 7f310f79b640 -1 rbd::mirror::LeaderWatcher: 0x558e6ffff200 handle_break_leader_lock: error breaking leader lock: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:08.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T15:20:08.338+0000 7fc685476640 -1 rbd::mirror::InstanceWatcher: 0x5557e35fbc08 handle_break_instance_lock: error breaking instance lock: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:08.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi195.stderr:2024-05-17T15:20:08.339+0000 7fd0d88b5640 -1 rbd::mirror::LeaderWatcher: 0x564c99414f00 handle_break_leader_lock: error breaking leader lock: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:08.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:20:08.339+0000 7f8ccc847640 -1 rbd::mirror::LeaderWatcher: 0x55b0fc5e7200 handle_break_leader_lock: error breaking leader lock: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:08.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T15:20:08.339+0000 7fc671c4f640 -1 rbd::mirror::InstanceWatcher: 0x5557e5e4ae08 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:08.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T15:20:08.339+0000 7fc67144e640 -1 rbd::mirror::InstanceWatcher: 0x5557e5e4b188 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:08.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T15:20:08.339+0000 7fc671c4f640 -1 rbd::mirror::InstanceWatcher: 0x5557e35fbc08 handle_remove_instance_object: error removing rbd_mirror_instance.5172 object: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:08.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T15:20:08.340+0000 7fc671c4f640 -1 rbd::mirror::InstanceWatcher: 0x5557e35fbc08 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:16.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:20:16.197+0000 7f8cae80b640 -1 rbd::mirror::PoolReplayer: 0x55b0fc318f00 list_mirroring_namespaces: failed to list namespaces: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:16.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:20:16.284+0000 7f8cd089f640 -1 rbd::mirror::Mirror: 0x55b0fb486120 update_pool_replayers: restarting blocklisted pool replayer for uuid: 0a5f21f5-898c-4c37-b257-500dcef76f47 cluster: cluster2 client: client.mirror 2024-05-17T15:20:16.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:20:16.285+0000 7f8cb9020640 -1 rbd::mirror::InstanceWatcher: 0x55b0fc8c0000 handle_remove_instance_object: error removing rbd_mirror_instance.4989 object: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:16.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T15:20:16.285+0000 7f8cb881f640 -1 rbd::mirror::InstanceWatcher: 0x55b0fc8c0000 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:18.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi195.stderr:2024-05-17T15:20:18.202+0000 7fd0ba879640 -1 rbd::mirror::PoolReplayer: 0x564c9901b800 list_mirroring_namespaces: failed to list namespaces: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:18.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi195.stderr:2024-05-17T15:20:18.260+0000 7fd0dce10640 -1 rbd::mirror::Mirror: 0x564c982d0120 update_pool_replayers: restarting blocklisted pool replayer for uuid: 0a5f21f5-898c-4c37-b257-500dcef76f47 cluster: cluster2 client: client.mirror 2024-05-17T15:20:18.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi195.stderr:2024-05-17T15:20:18.261+0000 7fd0c488d640 -1 rbd::mirror::InstanceWatcher: 0x564c9970c000 handle_remove_instance_object: error removing rbd_mirror_instance.5172 object: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:18.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi195.stderr:2024-05-17T15:20:18.261+0000 7fd0c508e640 -1 rbd::mirror::InstanceWatcher: 0x564c9970c000 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:18.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T15:20:18.339+0000 7f310ff9c640 -1 rbd::mirror::LeaderWatcher: 0x558e6ffff200 handle_acquire_leader_lock: error acquiring lock: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:18.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T15:20:18.339+0000 7f310f79b640 -1 rbd::mirror::LeaderWatcher: 0x558e6ffff200 handle_get_locker: error retrieving leader locker: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:23.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T15:20:23.218+0000 7f30f175f640 -1 rbd::mirror::PoolReplayer: 0x558e6fd34f00 list_mirroring_namespaces: failed to list namespaces: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:23.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T15:20:23.340+0000 7f310ff9c640 -1 rbd::mirror::LeaderWatcher: 0x558e6ffff200 handle_get_locker: error retrieving leader locker: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:23.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T15:20:23.496+0000 7f3114232640 -1 rbd::mirror::Mirror: 0x558e6ee88120 update_pool_replayers: restarting blocklisted pool replayer for uuid: 0a5f21f5-898c-4c37-b257-500dcef76f47 cluster: cluster2 client: client.mirror 2024-05-17T15:20:23.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T15:20:23.497+0000 7f30fb773640 -1 rbd::mirror::InstanceWatcher: 0x558e702b4000 handle_remove_instance_object: error removing rbd_mirror_instance.4743 object: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:20:23.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T15:20:23.497+0000 7f30fbf74640 -1 rbd::mirror::InstanceWatcher: 0x558e702b4000 handle_unregister_instance: error unregistering instance: (108) Cannot send after transport endpoint shutdown 2024-05-17T15:23:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T15:23:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ cleanup 110 2024-05-17T15:23:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local error_code=110 2024-05-17T15:23:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ set +e 2024-05-17T15:23:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 110 -ne 0 ']' 2024-05-17T15:23:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ status 2024-05-17T15:23:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster daemon image_pool image_ns image 2024-05-17T15:23:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2024-05-17T15:23:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 status' 2024-05-17T15:23:15.252 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 status 2024-05-17T15:23:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T15:23:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ ceph --cluster cluster1 -s 2024-05-17T15:23:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: cluster: 2024-05-17T15:23:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 5b29e51e-670a-453e-8d22-58d6f74d2947 2024-05-17T15:23:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: health: HEALTH_OK 2024-05-17T15:23:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: services: 2024-05-17T15:23:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mon: 1 daemons, quorum a (age 31m) 2024-05-17T15:23:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mgr: x(active, since 31m) 2024-05-17T15:23:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: osd: 3 osds: 3 up (since 31m), 3 in (since 31m) 2024-05-17T15:23:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: rbd-mirror: 4 daemons active (1 hosts) 2024-05-17T15:23:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: data: 2024-05-17T15:23:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: pools: 4 pools, 137 pgs 2024-05-17T15:23:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: objects: 1.14k objects, 3.3 GiB 2024-05-17T15:23:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: usage: 6.8 GiB used, 263 GiB / 270 GiB avail 2024-05-17T15:23:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: pgs: 137 active+clean 2024-05-17T15:23:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: io: 2024-05-17T15:23:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: client: 3.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s rd, 1 op/s wr 2024-05-17T15:23:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T15:23:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ ceph --cluster cluster1 service dump 2024-05-17T15:23:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:{ 2024-05-17T15:23:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "epoch": 30, 2024-05-17T15:23:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "modified": "2024-05-17T15:23:04.937471+0000", 2024-05-17T15:23:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "services": { 2024-05-17T15:23:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "rbd-mirror": { 2024-05-17T15:23:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "daemons": { 2024-05-17T15:23:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "summary": "", 2024-05-17T15:23:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "4737": { 2024-05-17T15:23:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "start_epoch": 12, 2024-05-17T15:23:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "start_stamp": "2024-05-17T15:05:22.975255+0000", 2024-05-17T15:23:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "gid": 4737, 2024-05-17T15:23:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "addr": "172.21.15.195:0/1409381439", 2024-05-17T15:23:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "metadata": { 2024-05-17T15:23:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "arch": "x86_64", 2024-05-17T15:23:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "ceph_release": "squid", 2024-05-17T15:23:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "ceph_version": "ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev)", 2024-05-17T15:23:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "ceph_version_short": "19.0.0-3735-gee59d86a", 2024-05-17T15:23:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-05-17T15:23:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "distro": "centos", 2024-05-17T15:23:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "distro_description": "CentOS Stream 9", 2024-05-17T15:23:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "distro_version": "9", 2024-05-17T15:23:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "hostname": "smithi195", 2024-05-17T15:23:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "id": "mirror.2", 2024-05-17T15:23:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "instance_id": "4737", 2024-05-17T15:23:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri May 3 16:10:18 UTC 2024", 2024-05-17T15:23:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "kernel_version": "5.14.0-446.el9.x86_64", 2024-05-17T15:23:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "mem_swap_kb": "0", 2024-05-17T15:23:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "mem_total_kb": "32488268", 2024-05-17T15:23:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "os": "Linux" 2024-05-17T15:23:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: }, 2024-05-17T15:23:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "task_status": {} 2024-05-17T15:23:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: }, 2024-05-17T15:23:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "4806": { 2024-05-17T15:23:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "start_epoch": 23, 2024-05-17T15:23:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "start_stamp": "2024-05-17T15:13:41.073013+0000", 2024-05-17T15:23:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "gid": 4806, 2024-05-17T15:23:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "addr": "172.21.15.195:0/3004032888", 2024-05-17T15:23:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "metadata": { 2024-05-17T15:23:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "arch": "x86_64", 2024-05-17T15:23:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "ceph_release": "squid", 2024-05-17T15:23:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "ceph_version": "ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev)", 2024-05-17T15:23:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "ceph_version_short": "19.0.0-3735-gee59d86a", 2024-05-17T15:23:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-05-17T15:23:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "distro": "centos", 2024-05-17T15:23:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "distro_description": "CentOS Stream 9", 2024-05-17T15:23:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "distro_version": "9", 2024-05-17T15:23:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "hostname": "smithi195", 2024-05-17T15:23:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "id": "mirror.0", 2024-05-17T15:23:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "instance_id": "4806", 2024-05-17T15:23:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri May 3 16:10:18 UTC 2024", 2024-05-17T15:23:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "kernel_version": "5.14.0-446.el9.x86_64", 2024-05-17T15:23:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "mem_swap_kb": "0", 2024-05-17T15:23:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "mem_total_kb": "32488268", 2024-05-17T15:23:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "os": "Linux" 2024-05-17T15:23:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: }, 2024-05-17T15:23:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "task_status": {} 2024-05-17T15:23:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: }, 2024-05-17T15:23:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "4983": { 2024-05-17T15:23:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "start_epoch": 25, 2024-05-17T15:23:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "start_stamp": "2024-05-17T15:15:16.069995+0000", 2024-05-17T15:23:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "gid": 4983, 2024-05-17T15:23:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "addr": "172.21.15.195:0/2339693579", 2024-05-17T15:23:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "metadata": { 2024-05-17T15:23:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "arch": "x86_64", 2024-05-17T15:23:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "ceph_release": "squid", 2024-05-17T15:23:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "ceph_version": "ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev)", 2024-05-17T15:23:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "ceph_version_short": "19.0.0-3735-gee59d86a", 2024-05-17T15:23:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-05-17T15:23:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "distro": "centos", 2024-05-17T15:23:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "distro_description": "CentOS Stream 9", 2024-05-17T15:23:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "distro_version": "9", 2024-05-17T15:23:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "hostname": "smithi195", 2024-05-17T15:23:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "id": "mirror.1", 2024-05-17T15:23:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "instance_id": "4983", 2024-05-17T15:23:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri May 3 16:10:18 UTC 2024", 2024-05-17T15:23:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "kernel_version": "5.14.0-446.el9.x86_64", 2024-05-17T15:23:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "mem_swap_kb": "0", 2024-05-17T15:23:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "mem_total_kb": "32488268", 2024-05-17T15:23:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "os": "Linux" 2024-05-17T15:23:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: }, 2024-05-17T15:23:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "task_status": {} 2024-05-17T15:23:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: }, 2024-05-17T15:23:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "5166": { 2024-05-17T15:23:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "start_epoch": 27, 2024-05-17T15:23:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "start_stamp": "2024-05-17T15:16:48.077023+0000", 2024-05-17T15:23:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "gid": 5166, 2024-05-17T15:23:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "addr": "172.21.15.195:0/928225161", 2024-05-17T15:23:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "metadata": { 2024-05-17T15:23:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "arch": "x86_64", 2024-05-17T15:23:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "ceph_release": "squid", 2024-05-17T15:23:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "ceph_version": "ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev)", 2024-05-17T15:23:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "ceph_version_short": "19.0.0-3735-gee59d86a", 2024-05-17T15:23:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-05-17T15:23:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "distro": "centos", 2024-05-17T15:23:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "distro_description": "CentOS Stream 9", 2024-05-17T15:23:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "distro_version": "9", 2024-05-17T15:23:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "hostname": "smithi195", 2024-05-17T15:23:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "id": "mirror.3", 2024-05-17T15:23:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "instance_id": "5166", 2024-05-17T15:23:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri May 3 16:10:18 UTC 2024", 2024-05-17T15:23:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "kernel_version": "5.14.0-446.el9.x86_64", 2024-05-17T15:23:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "mem_swap_kb": "0", 2024-05-17T15:23:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "mem_total_kb": "32488268", 2024-05-17T15:23:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "os": "Linux" 2024-05-17T15:23:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: }, 2024-05-17T15:23:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "task_status": {} 2024-05-17T15:23:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: } 2024-05-17T15:23:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: } 2024-05-17T15:23:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: } 2024-05-17T15:23:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: } 2024-05-17T15:23:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:} 2024-05-17T15:23:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T15:23:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ ceph --cluster cluster1 service status 2024-05-17T15:23:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:{ 2024-05-17T15:23:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "rbd-mirror": { 2024-05-17T15:23:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "4737": { 2024-05-17T15:23:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "status_stamp": "2024-05-17T15:22:57.987070+0000", 2024-05-17T15:23:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "last_beacon": "2024-05-17T15:23:12.987577+0000", 2024-05-17T15:23:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "status": { 2024-05-17T15:23:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":13,\"image_error_count\":0,\"image_local_count\":50,\"image_remote_count\":50,\"image_warning_count\":0,\"instance_id\":\"4740\",\"leader\":true,\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4743\"}}" 2024-05-17T15:23:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: } 2024-05-17T15:23:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: }, 2024-05-17T15:23:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "4806": { 2024-05-17T15:23:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "status_stamp": "2024-05-17T15:23:16.097751+0000", 2024-05-17T15:23:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "last_beacon": "2024-05-17T15:23:16.097751+0000", 2024-05-17T15:23:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "status": { 2024-05-17T15:23:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":13,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4809\"},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0,\"instance_id\":\"4811\"}}" 2024-05-17T15:23:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: } 2024-05-17T15:23:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: }, 2024-05-17T15:23:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "4983": { 2024-05-17T15:23:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "status_stamp": "2024-05-17T15:22:51.088941+0000", 2024-05-17T15:23:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "last_beacon": "2024-05-17T15:23:16.090081+0000", 2024-05-17T15:23:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "status": { 2024-05-17T15:23:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":12,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4986\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4989\"}}" 2024-05-17T15:23:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: } 2024-05-17T15:23:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: }, 2024-05-17T15:23:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "5166": { 2024-05-17T15:23:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "status_stamp": "2024-05-17T15:22:53.092957+0000", 2024-05-17T15:23:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "last_beacon": "2024-05-17T15:23:13.093697+0000", 2024-05-17T15:23:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "status": { 2024-05-17T15:23:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":12,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5169\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5172\"}}" 2024-05-17T15:23:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: } 2024-05-17T15:23:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: } 2024-05-17T15:23:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: } 2024-05-17T15:23:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:} 2024-05-17T15:23:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2024-05-17T15:23:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-05-17T15:23:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror images 2024-05-17T15:23:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror images' 2024-05-17T15:23:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' ls -l 2024-05-17T15:23:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:NAME SIZE PARENT FMT PROT LOCK 2024-05-17T15:23:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_1 128 MiB 2 excl 2024-05-17T15:23:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_1@snap 128 MiB 2 2024-05-17T15:23:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_10 128 MiB 2 excl 2024-05-17T15:23:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_10@snap 128 MiB 2 2024-05-17T15:23:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_11 128 MiB 2 excl 2024-05-17T15:23:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_11@snap 128 MiB 2 2024-05-17T15:23:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12 128 MiB 2 excl 2024-05-17T15:23:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12@snap 128 MiB 2 2024-05-17T15:23:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_13 128 MiB 2 excl 2024-05-17T15:23:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_13@snap 128 MiB 2 2024-05-17T15:23:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_14 128 MiB 2 excl 2024-05-17T15:23:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_14@snap 128 MiB 2 2024-05-17T15:23:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_15 128 MiB 2 excl 2024-05-17T15:23:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_15@snap 128 MiB 2 2024-05-17T15:23:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_16 128 MiB 2 excl 2024-05-17T15:23:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_16@snap 128 MiB 2 2024-05-17T15:23:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_17 128 MiB 2 excl 2024-05-17T15:23:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_17@snap 128 MiB 2 2024-05-17T15:23:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_18 128 MiB 2 excl 2024-05-17T15:23:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_18@snap 128 MiB 2 2024-05-17T15:23:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_19 128 MiB 2 excl 2024-05-17T15:23:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_19@snap 128 MiB 2 2024-05-17T15:23:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_2 128 MiB 2 excl 2024-05-17T15:23:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_2@snap 128 MiB 2 2024-05-17T15:23:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_20 128 MiB 2 excl 2024-05-17T15:23:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_20@snap 128 MiB 2 2024-05-17T15:23:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_21 128 MiB 2 excl 2024-05-17T15:23:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_21@snap 128 MiB 2 2024-05-17T15:23:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_22 128 MiB 2 excl 2024-05-17T15:23:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_22@snap 128 MiB 2 2024-05-17T15:23:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_23 128 MiB 2 excl 2024-05-17T15:23:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_23@snap 128 MiB 2 2024-05-17T15:23:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_24 128 MiB 2 excl 2024-05-17T15:23:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_24@snap 128 MiB 2 2024-05-17T15:23:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_25 128 MiB 2 excl 2024-05-17T15:23:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_25@snap 128 MiB 2 2024-05-17T15:23:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_26 128 MiB 2 excl 2024-05-17T15:23:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_26@snap 128 MiB 2 2024-05-17T15:23:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_27 128 MiB 2 excl 2024-05-17T15:23:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_28 128 MiB 2 excl 2024-05-17T15:23:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_29 128 MiB 2 excl 2024-05-17T15:23:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_3 128 MiB 2 excl 2024-05-17T15:23:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_3@snap 128 MiB 2 2024-05-17T15:23:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_30 128 MiB 2 excl 2024-05-17T15:23:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_31 128 MiB 2 excl 2024-05-17T15:23:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_32 128 MiB 2 excl 2024-05-17T15:23:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_33 128 MiB 2 excl 2024-05-17T15:23:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_34 128 MiB 2 excl 2024-05-17T15:23:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_35 128 MiB 2 excl 2024-05-17T15:23:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_36 128 MiB 2 excl 2024-05-17T15:23:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_37 128 MiB 2 excl 2024-05-17T15:23:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_38 128 MiB 2 excl 2024-05-17T15:23:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_39 128 MiB 2 excl 2024-05-17T15:23:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_4 128 MiB 2 excl 2024-05-17T15:23:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_4@snap 128 MiB 2 2024-05-17T15:23:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_40 128 MiB 2 excl 2024-05-17T15:23:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_41 128 MiB 2 excl 2024-05-17T15:23:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_42 128 MiB 2 excl 2024-05-17T15:23:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_43 128 MiB 2 excl 2024-05-17T15:23:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_44 128 MiB 2 excl 2024-05-17T15:23:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_45 128 MiB 2 excl 2024-05-17T15:23:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_46 128 MiB 2 excl 2024-05-17T15:23:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_47 128 MiB 2 excl 2024-05-17T15:23:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_48 128 MiB 2 excl 2024-05-17T15:23:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_49 128 MiB 2 excl 2024-05-17T15:23:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_5 128 MiB 2 excl 2024-05-17T15:23:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_5@snap 128 MiB 2 2024-05-17T15:23:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_50 128 MiB 2 excl 2024-05-17T15:23:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_6 128 MiB 2 excl 2024-05-17T15:23:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_6@snap 128 MiB 2 2024-05-17T15:23:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_7 128 MiB 2 excl 2024-05-17T15:23:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_7@snap 128 MiB 2 2024-05-17T15:23:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_8 128 MiB 2 excl 2024-05-17T15:23:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_8@snap 128 MiB 2 2024-05-17T15:23:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_9 128 MiB 2 excl 2024-05-17T15:23:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_9@snap 128 MiB 2 2024-05-17T15:23:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror mirror pool info 2024-05-17T15:23:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror mirror pool info' 2024-05-17T15:23:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' mirror pool info 2024-05-17T15:23:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mode: image 2024-05-17T15:23:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Site Name: cluster1 2024-05-17T15:23:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Peer Sites: 2024-05-17T15:23:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:UUID: 19bb688f-aeae-4d87-ba1c-7b8f4695256d 2024-05-17T15:23:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Name: cluster2 2024-05-17T15:23:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirror UUID: 7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Direction: rx-tx 2024-05-17T15:23:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Client: client.mirror 2024-05-17T15:23:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror mirror pool status 2024-05-17T15:23:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror mirror pool status' 2024-05-17T15:23:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T15:23:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' mirror pool status --verbose 2024-05-17T15:23:17.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:health: WARNING 2024-05-17T15:23:17.609 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:daemon health: OK 2024-05-17T15:23:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image health: WARNING 2024-05-17T15:23:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:images: 50 total 2024-05-17T15:23:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 13 unknown 2024-05-17T15:23:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 37 starting_replay 2024-05-17T15:23:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:DAEMONS 2024-05-17T15:23:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:service 4737: 2024-05-17T15:23:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: instance_id: 4740 2024-05-17T15:23:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: client_id: mirror.2 2024-05-17T15:23:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: hostname: smithi195 2024-05-17T15:23:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: version: 19.0.0-3735-gee59d86a 2024-05-17T15:23:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: leader: true 2024-05-17T15:23:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: health: OK 2024-05-17T15:23:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:service 4806: 2024-05-17T15:23:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: instance_id: 4809 2024-05-17T15:23:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: client_id: mirror.0 2024-05-17T15:23:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: hostname: smithi195 2024-05-17T15:23:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: version: 19.0.0-3735-gee59d86a 2024-05-17T15:23:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: leader: false 2024-05-17T15:23:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: health: OK 2024-05-17T15:23:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:service 4983: 2024-05-17T15:23:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: instance_id: 4986 2024-05-17T15:23:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: client_id: mirror.1 2024-05-17T15:23:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: hostname: smithi195 2024-05-17T15:23:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: version: 19.0.0-3735-gee59d86a 2024-05-17T15:23:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: leader: false 2024-05-17T15:23:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: health: OK 2024-05-17T15:23:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:service 5166: 2024-05-17T15:23:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: instance_id: 5169 2024-05-17T15:23:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: client_id: mirror.3 2024-05-17T15:23:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: hostname: smithi195 2024-05-17T15:23:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: version: 19.0.0-3735-gee59d86a 2024-05-17T15:23:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: leader: false 2024-05-17T15:23:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: health: OK 2024-05-17T15:23:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:IMAGES 2024-05-17T15:23:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_1: 2024-05-17T15:23:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: ceaa7752-4708-4453-aad7-8c9fe4b5ad62 2024-05-17T15:23:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958843,"remote_snapshot_timestamp":1715958843,"replay_state":"idle"} 2024-05-17T15:23:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.1 on smithi195 2024-05-17T15:23:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:16 2024-05-17T15:23:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_10: 2024-05-17T15:23:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 95950a54-837a-461f-aafc-36eac3b2a17a 2024-05-17T15:23:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958916,"remote_snapshot_timestamp":1715958916,"replay_state":"idle"} 2024-05-17T15:23:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.3 on smithi195 2024-05-17T15:23:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:48 2024-05-17T15:23:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_11: 2024-05-17T15:23:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 0e1e48a7-bf0e-42d2-9d09-07eeeeaaf56e 2024-05-17T15:23:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958940,"remote_snapshot_timestamp":1715958940,"replay_state":"idle"} 2024-05-17T15:23:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.1 on smithi195 2024-05-17T15:23:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:16 2024-05-17T15:23:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12: 2024-05-17T15:23:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: c50f08e5-ef2e-4cc9-afd7-fb18dd3f03dc 2024-05-17T15:23:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958948,"remote_snapshot_timestamp":1715958948,"replay_state":"idle"} 2024-05-17T15:23:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.1 on smithi195 2024-05-17T15:23:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:16 2024-05-17T15:23:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_13: 2024-05-17T15:23:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 76d62c45-9c33-4b38-aeb9-d12dc0c2247a 2024-05-17T15:23:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958955,"remote_snapshot_timestamp":1715958955,"replay_state":"idle"} 2024-05-17T15:23:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.0 on smithi195 2024-05-17T15:23:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:11 2024-05-17T15:23:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_14: 2024-05-17T15:23:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 9e41e805-f938-4777-a0dd-ce32b1acfcf1 2024-05-17T15:23:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958962,"remote_snapshot_timestamp":1715958962,"replay_state":"idle"} 2024-05-17T15:23:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.1 on smithi195 2024-05-17T15:23:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:16 2024-05-17T15:23:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_15: 2024-05-17T15:23:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: fb0ce489-3f67-44e1-971c-03b1005074a7 2024-05-17T15:23:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958969,"remote_snapshot_timestamp":1715958969,"replay_state":"idle"} 2024-05-17T15:23:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.0 on smithi195 2024-05-17T15:23:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:11 2024-05-17T15:23:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:56 2024-05-17T15:23:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_16: 2024-05-17T15:23:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 9cc94201-e324-4319-a646-d571d0e3df05 2024-05-17T15:23:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958977,"remote_snapshot_timestamp":1715958977,"replay_state":"idle"} 2024-05-17T15:23:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.2 on smithi195 2024-05-17T15:23:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:53 2024-05-17T15:23:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_17: 2024-05-17T15:23:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: bef15777-4cd7-47fb-8829-12dabc805b90 2024-05-17T15:23:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958985,"remote_snapshot_timestamp":1715958985,"replay_state":"idle"} 2024-05-17T15:23:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.1 on smithi195 2024-05-17T15:23:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:16 2024-05-17T15:23:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_18: 2024-05-17T15:23:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: e3330572-d02a-4104-a14f-d83973929a2c 2024-05-17T15:23:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958993,"remote_snapshot_timestamp":1715958993,"replay_state":"idle"} 2024-05-17T15:23:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.2 on smithi195 2024-05-17T15:23:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:53 2024-05-17T15:23:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:56 2024-05-17T15:23:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_19: 2024-05-17T15:23:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 3df98478-a489-4a33-b01c-132fa8241534 2024-05-17T15:23:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715959001,"remote_snapshot_timestamp":1715959001,"replay_state":"idle"} 2024-05-17T15:23:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.0 on smithi195 2024-05-17T15:23:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:11 2024-05-17T15:23:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_2: 2024-05-17T15:23:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 8d44a27c-5bdd-4a78-9b8a-9a4d22d34367 2024-05-17T15:23:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958851,"remote_snapshot_timestamp":1715958851,"replay_state":"idle"} 2024-05-17T15:23:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.0 on smithi195 2024-05-17T15:23:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:11 2024-05-17T15:23:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:56 2024-05-17T15:23:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_20: 2024-05-17T15:23:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 99995afa-10de-4385-a379-455ec2be8a34 2024-05-17T15:23:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":131973120.0,"last_snapshot_bytes":131973120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715959041,"remote_snapshot_timestamp":1715959041,"replay_state":"idle"} 2024-05-17T15:23:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.2 on smithi195 2024-05-17T15:23:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:53 2024-05-17T15:23:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: down+unknown 2024-05-17T15:23:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: status not found 2024-05-17T15:23:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17T15:23:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_21: 2024-05-17T15:23:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 7fede8a9-d664-48e9-b43a-740b49fc17be 2024-05-17T15:23:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134199296.0,"last_snapshot_bytes":134199296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715959048,"remote_snapshot_timestamp":1715959048,"replay_state":"idle"} 2024-05-17T15:23:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.1 on smithi195 2024-05-17T15:23:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:16 2024-05-17T15:23:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_22: 2024-05-17T15:23:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 7b475e11-a35f-4917-9f29-282df00e1098 2024-05-17T15:23:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134085632.0,"last_snapshot_bytes":134085632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715959055,"remote_snapshot_timestamp":1715959055,"replay_state":"idle"} 2024-05-17T15:23:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.3 on smithi195 2024-05-17T15:23:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:48 2024-05-17T15:23:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:56 2024-05-17T15:23:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_23: 2024-05-17T15:23:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 2968dbed-dc66-4e82-be5f-bc6e3322a6c1 2024-05-17T15:23:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134077440.0,"last_snapshot_bytes":134077440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715959063,"remote_snapshot_timestamp":1715959063,"replay_state":"idle"} 2024-05-17T15:23:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.0 on smithi195 2024-05-17T15:23:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:11 2024-05-17T15:23:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_24: 2024-05-17T15:23:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: d95d4f33-deca-446d-88ca-2fd4ef56f554 2024-05-17T15:23:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133702656.0,"last_snapshot_bytes":133702656,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715959071,"remote_snapshot_timestamp":1715959071,"replay_state":"idle"} 2024-05-17T15:23:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.2 on smithi195 2024-05-17T15:23:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:53 2024-05-17T15:23:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: down+unknown 2024-05-17T15:23:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: status not found 2024-05-17T15:23:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17T15:23:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_25: 2024-05-17T15:23:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 5f4f313b-d9df-420f-8c92-2d9e481184bc 2024-05-17T15:23:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133881856.0,"last_snapshot_bytes":133881856,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715959079,"remote_snapshot_timestamp":1715959079,"replay_state":"idle"} 2024-05-17T15:23:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.1 on smithi195 2024-05-17T15:23:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:16 2024-05-17T15:23:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:56 2024-05-17T15:23:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_26: 2024-05-17T15:23:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 9d50b403-8aa5-4da6-8e72-4a3f61c9358b 2024-05-17T15:23:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134135808.0,"last_snapshot_bytes":134135808,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715959087,"remote_snapshot_timestamp":1715959087,"replay_state":"idle"} 2024-05-17T15:23:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.3 on smithi195 2024-05-17T15:23:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:48 2024-05-17T15:23:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_27: 2024-05-17T15:23:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: d0b80867-2a3b-4947-ae2d-30cd6b4ea8cc 2024-05-17T15:23:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958565,"remote_snapshot_timestamp":1715958565,"replay_state":"idle"} 2024-05-17T15:23:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.3 on smithi195 2024-05-17T15:23:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:48 2024-05-17T15:23:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: down+unknown 2024-05-17T15:23:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: status not found 2024-05-17T15:23:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17T15:23:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_28: 2024-05-17T15:23:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 90f08d14-8ef0-4bff-9f0d-d804ed59aad1 2024-05-17T15:23:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958579,"remote_snapshot_timestamp":1715958579,"replay_state":"idle"} 2024-05-17T15:23:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.1 on smithi195 2024-05-17T15:23:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:16 2024-05-17T15:23:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:56 2024-05-17T15:23:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_29: 2024-05-17T15:23:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 59928e42-fdf9-4cdf-a1c9-1cabd73a05c5 2024-05-17T15:23:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958585,"remote_snapshot_timestamp":1715958585,"replay_state":"idle"} 2024-05-17T15:23:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.1 on smithi195 2024-05-17T15:23:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:16 2024-05-17T15:23:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:56 2024-05-17T15:23:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_3: 2024-05-17T15:23:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 5191746b-1578-4957-a477-41bde526d1e0 2024-05-17T15:23:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958859,"remote_snapshot_timestamp":1715958859,"replay_state":"idle"} 2024-05-17T15:23:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.2 on smithi195 2024-05-17T15:23:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:53 2024-05-17T15:23:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:56 2024-05-17T15:23:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_30: 2024-05-17T15:23:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 2031e3c2-db04-4d88-a9f5-2a8ddf1e9fce 2024-05-17T15:23:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958598,"remote_snapshot_timestamp":1715958598,"replay_state":"idle"} 2024-05-17T15:23:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.0 on smithi195 2024-05-17T15:23:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:11 2024-05-17T15:23:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_31: 2024-05-17T15:23:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 219fe1bd-503a-4377-b187-f83062f5b1ba 2024-05-17T15:23:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958611,"remote_snapshot_timestamp":1715958611,"replay_state":"idle"} 2024-05-17T15:23:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.0 on smithi195 2024-05-17T15:23:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:11 2024-05-17T15:23:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_32: 2024-05-17T15:23:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 0835fff5-1be2-449a-8ff6-85a5ed6b71f8 2024-05-17T15:23:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958626,"remote_snapshot_timestamp":1715958626,"replay_state":"idle"} 2024-05-17T15:23:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.0 on smithi195 2024-05-17T15:23:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:11 2024-05-17T15:23:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_33: 2024-05-17T15:23:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 0218530c-9285-4e1b-ac61-baf8583aceb9 2024-05-17T15:23:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958635,"remote_snapshot_timestamp":1715958635,"replay_state":"idle"} 2024-05-17T15:23:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.3 on smithi195 2024-05-17T15:23:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:48 2024-05-17T15:23:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_34: 2024-05-17T15:23:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 32af895d-8b5f-4d18-99db-b2ad45a7ac69 2024-05-17T15:23:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958642,"remote_snapshot_timestamp":1715958642,"replay_state":"idle"} 2024-05-17T15:23:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.3 on smithi195 2024-05-17T15:23:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:48 2024-05-17T15:23:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:03 2024-05-17T15:23:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_35: 2024-05-17T15:23:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 8c639172-6810-4a20-a5ee-19b5a50ec6fd 2024-05-17T15:23:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958653,"remote_snapshot_timestamp":1715958653,"replay_state":"idle"} 2024-05-17T15:23:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.2 on smithi195 2024-05-17T15:23:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:53 2024-05-17T15:23:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: down+unknown 2024-05-17T15:23:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: status not found 2024-05-17T15:23:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17T15:23:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_36: 2024-05-17T15:23:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: b8165b54-ecda-4ca3-8d0a-f0c1a85112c7 2024-05-17T15:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958667,"remote_snapshot_timestamp":1715958667,"replay_state":"idle"} 2024-05-17T15:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.2 on smithi195 2024-05-17T15:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:53 2024-05-17T15:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:56 2024-05-17T15:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_37: 2024-05-17T15:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 1cbb263d-bbcf-4037-8f80-52c0b1dfd284 2024-05-17T15:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958682,"remote_snapshot_timestamp":1715958682,"replay_state":"idle"} 2024-05-17T15:23:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.3 on smithi195 2024-05-17T15:23:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:48 2024-05-17T15:23:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: down+unknown 2024-05-17T15:23:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: status not found 2024-05-17T15:23:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17T15:23:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_38: 2024-05-17T15:23:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: f990d936-a771-483b-8728-fd1503201136 2024-05-17T15:23:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958691,"remote_snapshot_timestamp":1715958691,"replay_state":"idle"} 2024-05-17T15:23:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.2 on smithi195 2024-05-17T15:23:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:53 2024-05-17T15:23:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: down+unknown 2024-05-17T15:23:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: status not found 2024-05-17T15:23:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17T15:23:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_39: 2024-05-17T15:23:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: aae21142-bf9f-4d97-9ad7-8e4679ee037a 2024-05-17T15:23:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958703,"remote_snapshot_timestamp":1715958703,"replay_state":"idle"} 2024-05-17T15:23:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.3 on smithi195 2024-05-17T15:23:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:48 2024-05-17T15:23:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: down+unknown 2024-05-17T15:23:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: status not found 2024-05-17T15:23:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17T15:23:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_4: 2024-05-17T15:23:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 1fda3480-fa61-4b4b-93a6-2dd063ac50f4 2024-05-17T15:23:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958867,"remote_snapshot_timestamp":1715958867,"replay_state":"idle"} 2024-05-17T15:23:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.1 on smithi195 2024-05-17T15:23:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:16 2024-05-17T15:23:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_40: 2024-05-17T15:23:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 4abdb521-afae-4334-ad13-70815420eba8 2024-05-17T15:23:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958709,"remote_snapshot_timestamp":1715958709,"replay_state":"idle"} 2024-05-17T15:23:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.3 on smithi195 2024-05-17T15:23:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:48 2024-05-17T15:23:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:03 2024-05-17T15:23:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_41: 2024-05-17T15:23:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 69d8440b-93d1-40e8-b675-4ab5105af9bd 2024-05-17T15:23:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958721,"remote_snapshot_timestamp":1715958721,"replay_state":"idle"} 2024-05-17T15:23:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.3 on smithi195 2024-05-17T15:23:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:48 2024-05-17T15:23:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: down+unknown 2024-05-17T15:23:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: status not found 2024-05-17T15:23:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17T15:23:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_42: 2024-05-17T15:23:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 46ae8559-1830-4c3d-adfc-c8f96345efad 2024-05-17T15:23:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958729,"remote_snapshot_timestamp":1715958729,"replay_state":"idle"} 2024-05-17T15:23:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.2 on smithi195 2024-05-17T15:23:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:53 2024-05-17T15:23:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_43: 2024-05-17T15:23:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 414f7604-42d0-4537-bea3-bfcfbe66e210 2024-05-17T15:23:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958736,"remote_snapshot_timestamp":1715958736,"replay_state":"idle"} 2024-05-17T15:23:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.1 on smithi195 2024-05-17T15:23:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:16 2024-05-17T15:23:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_44: 2024-05-17T15:23:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 42d7dd77-f5e1-4c19-a790-e2073f5b67ee 2024-05-17T15:23:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958742,"remote_snapshot_timestamp":1715958742,"replay_state":"idle"} 2024-05-17T15:23:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.0 on smithi195 2024-05-17T15:23:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:11 2024-05-17T15:23:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:03 2024-05-17T15:23:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_45: 2024-05-17T15:23:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 729f0874-9d81-4812-83cf-ae9935d33036 2024-05-17T15:23:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958749,"remote_snapshot_timestamp":1715958749,"replay_state":"idle"} 2024-05-17T15:23:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.3 on smithi195 2024-05-17T15:23:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:48 2024-05-17T15:23:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: down+unknown 2024-05-17T15:23:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: status not found 2024-05-17T15:23:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17T15:23:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_46: 2024-05-17T15:23:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 36a3913d-378a-412a-ad5b-9c269860173e 2024-05-17T15:23:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958764,"remote_snapshot_timestamp":1715958764,"replay_state":"idle"} 2024-05-17T15:23:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.2 on smithi195 2024-05-17T15:23:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:53 2024-05-17T15:23:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_47: 2024-05-17T15:23:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 84e9e05b-5031-4201-93ab-915cf8a3c889 2024-05-17T15:23:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958770,"remote_snapshot_timestamp":1715958770,"replay_state":"idle"} 2024-05-17T15:23:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.2 on smithi195 2024-05-17T15:23:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:53 2024-05-17T15:23:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_48: 2024-05-17T15:23:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 84523fc1-2802-4868-9b68-13d10e9be7e2 2024-05-17T15:23:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958780,"remote_snapshot_timestamp":1715958780,"replay_state":"idle"} 2024-05-17T15:23:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.2 on smithi195 2024-05-17T15:23:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:53 2024-05-17T15:23:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:56 2024-05-17T15:23:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_49: 2024-05-17T15:23:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 6c86503a-19c3-470e-81c4-44311421ecfd 2024-05-17T15:23:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958792,"remote_snapshot_timestamp":1715958792,"replay_state":"idle"} 2024-05-17T15:23:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.3 on smithi195 2024-05-17T15:23:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:48 2024-05-17T15:23:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: down+unknown 2024-05-17T15:23:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: status not found 2024-05-17T15:23:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17T15:23:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_5: 2024-05-17T15:23:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 82562a71-d364-4946-b5e9-67440f1071c4 2024-05-17T15:23:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958876,"remote_snapshot_timestamp":1715958876,"replay_state":"idle"} 2024-05-17T15:23:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.0 on smithi195 2024-05-17T15:23:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:11 2024-05-17T15:23:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: down+unknown 2024-05-17T15:23:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: status not found 2024-05-17T15:23:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17T15:23:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_50: 2024-05-17T15:23:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 9ca9b16d-6fb0-46d4-9998-208fa4980f4f 2024-05-17T15:23:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958802,"remote_snapshot_timestamp":1715958802,"replay_state":"idle"} 2024-05-17T15:23:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.2 on smithi195 2024-05-17T15:23:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:53 2024-05-17T15:23:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_6: 2024-05-17T15:23:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: b40480e1-3aef-46bb-a86f-05a1ee0d2d26 2024-05-17T15:23:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958884,"remote_snapshot_timestamp":1715958884,"replay_state":"idle"} 2024-05-17T15:23:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.1 on smithi195 2024-05-17T15:23:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:16 2024-05-17T15:23:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: down+unknown 2024-05-17T15:23:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: status not found 2024-05-17T15:23:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17T15:23:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_7: 2024-05-17T15:23:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 6e4b54d1-2cee-4ccb-aef6-8e1d0ba0e670 2024-05-17T15:23:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958892,"remote_snapshot_timestamp":1715958892,"replay_state":"idle"} 2024-05-17T15:23:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.0 on smithi195 2024-05-17T15:23:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:11 2024-05-17T15:23:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_8: 2024-05-17T15:23:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 75a0868a-edaf-438e-a7e1-fedf558dccfa 2024-05-17T15:23:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958900,"remote_snapshot_timestamp":1715958900,"replay_state":"idle"} 2024-05-17T15:23:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.0 on smithi195 2024-05-17T15:23:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:11 2024-05-17T15:23:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+starting_replay 2024-05-17T15:23:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: starting replay 2024-05-17T15:23:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:22:51 2024-05-17T15:23:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_9: 2024-05-17T15:23:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: global_id: 6c3aab86-8ad4-4a10-8f48-3d66ec7f5b30 2024-05-17T15:23:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: up+replaying 2024-05-17T15:23:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1715958908,"remote_snapshot_timestamp":1715958908,"replay_state":"idle"} 2024-05-17T15:23:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: service: mirror.0 on smithi195 2024-05-17T15:23:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17 15:23:11 2024-05-17T15:23:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: peer_sites: 2024-05-17T15:23:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: name: cluster2 2024-05-17T15:23:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: state: down+unknown 2024-05-17T15:23:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: description: status not found 2024-05-17T15:23:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: last_update: 2024-05-17T15:23:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' ls 2024-05-17T15:23:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_1 info 2024-05-17T15:23:17.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:17.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_1 info' 2024-05-17T15:23:17.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_1 2024-05-17T15:23:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_1': 2024-05-17T15:23:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10c8f0e8d377 2024-05-17T15:23:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10c8f0e8d377 2024-05-17T15:23:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:04:43 2024 2024-05-17T15:23:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:14:08 2024 2024-05-17T15:23:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:04:43 2024 2024-05-17T15:23:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: ceaa7752-4708-4453-aad7-8c9fe4b5ad62 2024-05-17T15:23:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_1 journal status 2024-05-17T15:23:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_1 journal status' 2024-05-17T15:23:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_1 2024-05-17T15:23:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_1 2024-05-17T15:23:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_1 snapshots 2024-05-17T15:23:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_1 snapshots' 2024-05-17T15:23:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_1 2024-05-17T15:23:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 85 snap 128 MiB Fri May 17 15:14:04 2024 user 2024-05-17T15:23:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 86 .mirror.non_primary.ceaa7752-4708-4453-aad7-8c9fe4b5ad62.c474808e-dd3f-4410-a102-867322c16abd 128 MiB Fri May 17 15:14:05 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:86 copied) 2024-05-17T15:23:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_10 info 2024-05-17T15:23:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_10 info' 2024-05-17T15:23:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_10 2024-05-17T15:23:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_10': 2024-05-17T15:23:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1135c02e8df3 2024-05-17T15:23:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1135c02e8df3 2024-05-17T15:23:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:06:11 2024 2024-05-17T15:23:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:15:21 2024 2024-05-17T15:23:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:06:11 2024 2024-05-17T15:23:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 95950a54-837a-461f-aafc-36eac3b2a17a 2024-05-17T15:23:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_10 journal status 2024-05-17T15:23:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_10 journal status' 2024-05-17T15:23:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_10 2024-05-17T15:23:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_10 2024-05-17T15:23:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_10 snapshots 2024-05-17T15:23:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_10 snapshots' 2024-05-17T15:23:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_10 2024-05-17T15:23:18.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:18.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 103 snap 128 MiB Fri May 17 15:15:17 2024 user 2024-05-17T15:23:18.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 104 .mirror.non_primary.95950a54-837a-461f-aafc-36eac3b2a17a.6792e3ab-894c-4a3f-a22a-05615f94a0c7 128 MiB Fri May 17 15:15:18 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:104 copied) 2024-05-17T15:23:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_11 info' 2024-05-17T15:23:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_11 info 2024-05-17T15:23:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_11 2024-05-17T15:23:18.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_11': 2024-05-17T15:23:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 121ddc84b4e9 2024-05-17T15:23:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.121ddc84b4e9 2024-05-17T15:23:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:06:17 2024 2024-05-17T15:23:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:15:45 2024 2024-05-17T15:23:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:06:17 2024 2024-05-17T15:23:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 0e1e48a7-bf0e-42d2-9d09-07eeeeaaf56e 2024-05-17T15:23:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_11 journal status 2024-05-17T15:23:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_11 journal status' 2024-05-17T15:23:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_11 2024-05-17T15:23:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_11 2024-05-17T15:23:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_11 snapshots 2024-05-17T15:23:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_11 snapshots' 2024-05-17T15:23:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_11 2024-05-17T15:23:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 105 snap 128 MiB Fri May 17 15:15:41 2024 user 2024-05-17T15:23:18.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 106 .mirror.non_primary.0e1e48a7-bf0e-42d2-9d09-07eeeeaaf56e.b6233410-cd05-4216-a870-ac6680ac690f 128 MiB Fri May 17 15:15:42 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:106 copied) 2024-05-17T15:23:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_12 info 2024-05-17T15:23:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_12 info' 2024-05-17T15:23:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_12 2024-05-17T15:23:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_12': 2024-05-17T15:23:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1284a2612bb4 2024-05-17T15:23:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1284a2612bb4 2024-05-17T15:23:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:06:31 2024 2024-05-17T15:23:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:15:51 2024 2024-05-17T15:23:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:06:31 2024 2024-05-17T15:23:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: c50f08e5-ef2e-4cc9-afd7-fb18dd3f03dc 2024-05-17T15:23:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_12 journal status 2024-05-17T15:23:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_12 journal status' 2024-05-17T15:23:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_12 2024-05-17T15:23:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_12 2024-05-17T15:23:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_12 snapshots 2024-05-17T15:23:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_12 snapshots' 2024-05-17T15:23:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_12 2024-05-17T15:23:18.315 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:18.315 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 107 snap 128 MiB Fri May 17 15:15:49 2024 user 2024-05-17T15:23:18.315 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 108 .mirror.non_primary.c50f08e5-ef2e-4cc9-afd7-fb18dd3f03dc.aba25193-eb5c-4307-b3a3-bdacdaeb24e3 128 MiB Fri May 17 15:15:50 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:108 copied) 2024-05-17T15:23:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_13 info 2024-05-17T15:23:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_13 info' 2024-05-17T15:23:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_13 2024-05-17T15:23:18.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_13': 2024-05-17T15:23:18.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:18.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:18.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:18.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d3d5590995 2024-05-17T15:23:18.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d3d5590995 2024-05-17T15:23:18.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:18.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:18.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:06:38 2024 2024-05-17T15:23:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:15:58 2024 2024-05-17T15:23:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:06:38 2024 2024-05-17T15:23:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 76d62c45-9c33-4b38-aeb9-d12dc0c2247a 2024-05-17T15:23:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:18.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.362 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_13 journal status 2024-05-17T15:23:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_13 journal status' 2024-05-17T15:23:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_13 2024-05-17T15:23:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_13 2024-05-17T15:23:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_13 snapshots 2024-05-17T15:23:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_13 snapshots' 2024-05-17T15:23:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_13 2024-05-17T15:23:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 109 snap 128 MiB Fri May 17 15:15:56 2024 user 2024-05-17T15:23:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 110 .mirror.non_primary.76d62c45-9c33-4b38-aeb9-d12dc0c2247a.0029c9bb-8e94-43c3-949d-87add18fb96f 128 MiB Fri May 17 15:15:57 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:110 copied) 2024-05-17T15:23:18.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.455 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_14 info 2024-05-17T15:23:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_14 info' 2024-05-17T15:23:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_14 2024-05-17T15:23:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_14': 2024-05-17T15:23:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1135b7eb2bb 2024-05-17T15:23:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1135b7eb2bb 2024-05-17T15:23:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:06:49 2024 2024-05-17T15:23:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:16:05 2024 2024-05-17T15:23:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:06:49 2024 2024-05-17T15:23:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 9e41e805-f938-4777-a0dd-ce32b1acfcf1 2024-05-17T15:23:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_14 journal status 2024-05-17T15:23:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_14 journal status' 2024-05-17T15:23:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_14 2024-05-17T15:23:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_14 2024-05-17T15:23:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_14 snapshots 2024-05-17T15:23:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_14 snapshots' 2024-05-17T15:23:18.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_14 2024-05-17T15:23:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 111 snap 128 MiB Fri May 17 15:16:03 2024 user 2024-05-17T15:23:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 112 .mirror.non_primary.9e41e805-f938-4777-a0dd-ce32b1acfcf1.ec984273-63eb-40d3-b62b-f5bcb6066ab4 128 MiB Fri May 17 15:16:04 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:112 copied) 2024-05-17T15:23:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_15 info 2024-05-17T15:23:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_15 info' 2024-05-17T15:23:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_15 2024-05-17T15:23:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_15': 2024-05-17T15:23:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1293dc4d5196 2024-05-17T15:23:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1293dc4d5196 2024-05-17T15:23:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:07:22 2024 2024-05-17T15:23:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:16:13 2024 2024-05-17T15:23:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:07:22 2024 2024-05-17T15:23:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: fb0ce489-3f67-44e1-971c-03b1005074a7 2024-05-17T15:23:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_15 journal status 2024-05-17T15:23:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_15 journal status' 2024-05-17T15:23:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_15 2024-05-17T15:23:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_15 2024-05-17T15:23:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_15 snapshots 2024-05-17T15:23:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_15 snapshots' 2024-05-17T15:23:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_15 2024-05-17T15:23:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 113 snap 128 MiB Fri May 17 15:16:10 2024 user 2024-05-17T15:23:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 114 .mirror.non_primary.fb0ce489-3f67-44e1-971c-03b1005074a7.3c54a316-9bf2-4ba3-8a56-f36506513857 128 MiB Fri May 17 15:16:11 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:114 copied) 2024-05-17T15:23:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_16 info 2024-05-17T15:23:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_16 info' 2024-05-17T15:23:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_16 2024-05-17T15:23:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_16': 2024-05-17T15:23:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1293779945bc 2024-05-17T15:23:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1293779945bc 2024-05-17T15:23:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:07:10 2024 2024-05-17T15:23:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:16:22 2024 2024-05-17T15:23:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:07:10 2024 2024-05-17T15:23:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 9cc94201-e324-4319-a646-d571d0e3df05 2024-05-17T15:23:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_16 journal status 2024-05-17T15:23:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_16 journal status' 2024-05-17T15:23:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_16 2024-05-17T15:23:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_16 2024-05-17T15:23:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_16 snapshots 2024-05-17T15:23:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_16 snapshots' 2024-05-17T15:23:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_16 2024-05-17T15:23:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 115 snap 128 MiB Fri May 17 15:16:18 2024 user 2024-05-17T15:23:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 116 .mirror.non_primary.9cc94201-e324-4319-a646-d571d0e3df05.0a7fd2ae-6cd9-4ca3-9faf-bfd418b1846e 128 MiB Fri May 17 15:16:19 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:116 copied) 2024-05-17T15:23:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_17 info 2024-05-17T15:23:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_17 info' 2024-05-17T15:23:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_17 2024-05-17T15:23:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_17': 2024-05-17T15:23:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d31974d5da 2024-05-17T15:23:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d31974d5da 2024-05-17T15:23:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:07:23 2024 2024-05-17T15:23:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:16:30 2024 2024-05-17T15:23:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:07:23 2024 2024-05-17T15:23:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: bef15777-4cd7-47fb-8829-12dabc805b90 2024-05-17T15:23:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_17 journal status 2024-05-17T15:23:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_17 journal status' 2024-05-17T15:23:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_17 2024-05-17T15:23:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_17 2024-05-17T15:23:18.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_17 snapshots 2024-05-17T15:23:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_17 snapshots' 2024-05-17T15:23:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_17 2024-05-17T15:23:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 117 snap 128 MiB Fri May 17 15:16:26 2024 user 2024-05-17T15:23:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 118 .mirror.non_primary.bef15777-4cd7-47fb-8829-12dabc805b90.df393b09-cfd3-40b8-a678-8d210faf60d1 128 MiB Fri May 17 15:16:27 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:118 copied) 2024-05-17T15:23:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_18 info 2024-05-17T15:23:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_18 info' 2024-05-17T15:23:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_18 2024-05-17T15:23:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_18': 2024-05-17T15:23:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1135c4c03b23 2024-05-17T15:23:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1135c4c03b23 2024-05-17T15:23:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:07:38 2024 2024-05-17T15:23:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:16:38 2024 2024-05-17T15:23:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:07:38 2024 2024-05-17T15:23:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: e3330572-d02a-4104-a14f-d83973929a2c 2024-05-17T15:23:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_18 journal status 2024-05-17T15:23:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_18 journal status' 2024-05-17T15:23:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_18 2024-05-17T15:23:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_18 2024-05-17T15:23:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_18 snapshots 2024-05-17T15:23:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_18 snapshots' 2024-05-17T15:23:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_18 2024-05-17T15:23:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 119 snap 128 MiB Fri May 17 15:16:34 2024 user 2024-05-17T15:23:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 120 .mirror.non_primary.e3330572-d02a-4104-a14f-d83973929a2c.7d07ae48-378d-44e7-8e3e-0029a0f7ecb2 128 MiB Fri May 17 15:16:35 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:120 copied) 2024-05-17T15:23:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_19 info 2024-05-17T15:23:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_19 info' 2024-05-17T15:23:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_19 2024-05-17T15:23:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_19': 2024-05-17T15:23:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 128479b7028b 2024-05-17T15:23:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.128479b7028b 2024-05-17T15:23:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:07:50 2024 2024-05-17T15:23:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:16:44 2024 2024-05-17T15:23:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:07:50 2024 2024-05-17T15:23:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 3df98478-a489-4a33-b01c-132fa8241534 2024-05-17T15:23:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_19 journal status 2024-05-17T15:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_19 journal status' 2024-05-17T15:23:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_19 2024-05-17T15:23:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_19 2024-05-17T15:23:19.222 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_19 snapshots 2024-05-17T15:23:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_19 snapshots' 2024-05-17T15:23:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_19 2024-05-17T15:23:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 121 snap 128 MiB Fri May 17 15:16:42 2024 user 2024-05-17T15:23:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 122 .mirror.non_primary.3df98478-a489-4a33-b01c-132fa8241534.e5d331bc-fc90-4efe-a19e-3d27c84cb35e 128 MiB Fri May 17 15:16:43 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:122 copied) 2024-05-17T15:23:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_2 info' 2024-05-17T15:23:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_2 info 2024-05-17T15:23:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_2 2024-05-17T15:23:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_2': 2024-05-17T15:23:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d3b6d76de2 2024-05-17T15:23:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d3b6d76de2 2024-05-17T15:23:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:04:54 2024 2024-05-17T15:23:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:14:16 2024 2024-05-17T15:23:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:04:54 2024 2024-05-17T15:23:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 8d44a27c-5bdd-4a78-9b8a-9a4d22d34367 2024-05-17T15:23:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_2 journal status 2024-05-17T15:23:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_2 journal status' 2024-05-17T15:23:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_2 2024-05-17T15:23:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_2 2024-05-17T15:23:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_2 snapshots 2024-05-17T15:23:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_2 snapshots' 2024-05-17T15:23:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_2 2024-05-17T15:23:19.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 87 snap 128 MiB Fri May 17 15:14:12 2024 user 2024-05-17T15:23:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 88 .mirror.non_primary.8d44a27c-5bdd-4a78-9b8a-9a4d22d34367.7bc83e6b-bb5a-40c8-949f-deb407b08a39 128 MiB Fri May 17 15:14:13 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:88 copied) 2024-05-17T15:23:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_20 info 2024-05-17T15:23:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_20 info' 2024-05-17T15:23:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_20 2024-05-17T15:23:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_20': 2024-05-17T15:23:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12939310b8e8 2024-05-17T15:23:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12939310b8e8 2024-05-17T15:23:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:08:01 2024 2024-05-17T15:23:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:17:24 2024 2024-05-17T15:23:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:08:01 2024 2024-05-17T15:23:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 99995afa-10de-4385-a379-455ec2be8a34 2024-05-17T15:23:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_20 journal status 2024-05-17T15:23:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_20 journal status' 2024-05-17T15:23:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_20 2024-05-17T15:23:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_20 2024-05-17T15:23:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_20 snapshots 2024-05-17T15:23:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_20 snapshots' 2024-05-17T15:23:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_20 2024-05-17T15:23:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 123 snap 128 MiB Fri May 17 15:17:22 2024 user 2024-05-17T15:23:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 124 .mirror.non_primary.99995afa-10de-4385-a379-455ec2be8a34.8f009ea8-7a2b-4ad2-8cae-36473aab1600 128 MiB Fri May 17 15:17:23 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:124 copied) 2024-05-17T15:23:19.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_21 info 2024-05-17T15:23:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_21 info' 2024-05-17T15:23:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_21 2024-05-17T15:23:19.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_21': 2024-05-17T15:23:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d3ade11e6e 2024-05-17T15:23:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d3ade11e6e 2024-05-17T15:23:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:08:09 2024 2024-05-17T15:23:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:17:31 2024 2024-05-17T15:23:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:08:09 2024 2024-05-17T15:23:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 7fede8a9-d664-48e9-b43a-740b49fc17be 2024-05-17T15:23:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_21 journal status 2024-05-17T15:23:19.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_21 journal status' 2024-05-17T15:23:19.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_21 2024-05-17T15:23:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_21 2024-05-17T15:23:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_21 snapshots 2024-05-17T15:23:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_21 snapshots' 2024-05-17T15:23:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_21 2024-05-17T15:23:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 125 snap 128 MiB Fri May 17 15:17:29 2024 user 2024-05-17T15:23:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 126 .mirror.non_primary.7fede8a9-d664-48e9-b43a-740b49fc17be.5f523d51-06f2-4582-8759-b8934deaf81f 128 MiB Fri May 17 15:17:30 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:126 copied) 2024-05-17T15:23:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_22 info 2024-05-17T15:23:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_22 info' 2024-05-17T15:23:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_22 2024-05-17T15:23:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_22': 2024-05-17T15:23:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 113594126061 2024-05-17T15:23:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.113594126061 2024-05-17T15:23:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:08:23 2024 2024-05-17T15:23:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:17:39 2024 2024-05-17T15:23:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:08:23 2024 2024-05-17T15:23:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 7b475e11-a35f-4917-9f29-282df00e1098 2024-05-17T15:23:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_22 journal status 2024-05-17T15:23:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_22 journal status' 2024-05-17T15:23:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_22 2024-05-17T15:23:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_22 2024-05-17T15:23:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_22 snapshots 2024-05-17T15:23:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_22 snapshots' 2024-05-17T15:23:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_22 2024-05-17T15:23:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 127 snap 128 MiB Fri May 17 15:17:36 2024 user 2024-05-17T15:23:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 128 .mirror.non_primary.7b475e11-a35f-4917-9f29-282df00e1098.e9572fb4-d65f-4acd-87ad-5c408b158da6 128 MiB Fri May 17 15:17:37 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:128 copied) 2024-05-17T15:23:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_23 info 2024-05-17T15:23:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_23 info' 2024-05-17T15:23:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_23 2024-05-17T15:23:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_23': 2024-05-17T15:23:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1284c16da516 2024-05-17T15:23:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1284c16da516 2024-05-17T15:23:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:08:33 2024 2024-05-17T15:23:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:17:47 2024 2024-05-17T15:23:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:08:33 2024 2024-05-17T15:23:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 2968dbed-dc66-4e82-be5f-bc6e3322a6c1 2024-05-17T15:23:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_23 journal status' 2024-05-17T15:23:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_23 journal status 2024-05-17T15:23:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_23 2024-05-17T15:23:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_23 2024-05-17T15:23:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_23 snapshots 2024-05-17T15:23:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_23 snapshots' 2024-05-17T15:23:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_23 2024-05-17T15:23:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 129 snap 128 MiB Fri May 17 15:17:44 2024 user 2024-05-17T15:23:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 130 .mirror.non_primary.2968dbed-dc66-4e82-be5f-bc6e3322a6c1.b4b492ca-e3b2-4272-aff3-60c2973b4c92 128 MiB Fri May 17 15:17:45 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:130 copied) 2024-05-17T15:23:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_24 info 2024-05-17T15:23:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_24 info' 2024-05-17T15:23:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_24 2024-05-17T15:23:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_24': 2024-05-17T15:23:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12a3ab3a6e1a 2024-05-17T15:23:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12a3ab3a6e1a 2024-05-17T15:23:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:08:48 2024 2024-05-17T15:23:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:17:56 2024 2024-05-17T15:23:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:08:48 2024 2024-05-17T15:23:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: d95d4f33-deca-446d-88ca-2fd4ef56f554 2024-05-17T15:23:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_24 journal status 2024-05-17T15:23:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_24 journal status' 2024-05-17T15:23:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_24 2024-05-17T15:23:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_24 2024-05-17T15:23:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_24 snapshots 2024-05-17T15:23:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_24 snapshots' 2024-05-17T15:23:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_24 2024-05-17T15:23:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 131 snap 128 MiB Fri May 17 15:17:52 2024 user 2024-05-17T15:23:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 132 .mirror.non_primary.d95d4f33-deca-446d-88ca-2fd4ef56f554.647c51a4-57c2-400f-8e98-4cbebaeefe79 128 MiB Fri May 17 15:17:53 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:132 copied) 2024-05-17T15:23:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_25 info 2024-05-17T15:23:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_25 info' 2024-05-17T15:23:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_25 2024-05-17T15:23:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_25': 2024-05-17T15:23:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12a3ba5aa645 2024-05-17T15:23:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12a3ba5aa645 2024-05-17T15:23:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:08:59 2024 2024-05-17T15:23:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:18:04 2024 2024-05-17T15:23:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:08:59 2024 2024-05-17T15:23:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 5f4f313b-d9df-420f-8c92-2d9e481184bc 2024-05-17T15:23:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_25 journal status 2024-05-17T15:23:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_25 journal status' 2024-05-17T15:23:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_25 2024-05-17T15:23:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_25 2024-05-17T15:23:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_25 snapshots 2024-05-17T15:23:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_25 snapshots' 2024-05-17T15:23:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_25 2024-05-17T15:23:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 133 snap 128 MiB Fri May 17 15:18:00 2024 user 2024-05-17T15:23:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 134 .mirror.non_primary.5f4f313b-d9df-420f-8c92-2d9e481184bc.b9070009-8f92-4ab0-ab4a-05765f77b0db 128 MiB Fri May 17 15:18:01 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:134 copied) 2024-05-17T15:23:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_26 info' 2024-05-17T15:23:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_26 info 2024-05-17T15:23:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_26 2024-05-17T15:23:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_26': 2024-05-17T15:23:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 11356b06411f 2024-05-17T15:23:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.11356b06411f 2024-05-17T15:23:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:09:13 2024 2024-05-17T15:23:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:18:12 2024 2024-05-17T15:23:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:09:13 2024 2024-05-17T15:23:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 9d50b403-8aa5-4da6-8e72-4a3f61c9358b 2024-05-17T15:23:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_26 journal status 2024-05-17T15:23:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_26 journal status' 2024-05-17T15:23:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_26 2024-05-17T15:23:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_26 2024-05-17T15:23:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_26 snapshots 2024-05-17T15:23:20.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_26 snapshots' 2024-05-17T15:23:20.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_26 2024-05-17T15:23:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 135 snap 128 MiB Fri May 17 15:18:08 2024 user 2024-05-17T15:23:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 136 .mirror.non_primary.9d50b403-8aa5-4da6-8e72-4a3f61c9358b.9be1aced-5fba-4aa6-acc8-32d4c7b68f17 128 MiB Fri May 17 15:18:09 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:136 copied) 2024-05-17T15:23:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_27 info' 2024-05-17T15:23:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_27 info 2024-05-17T15:23:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_27 2024-05-17T15:23:20.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_27': 2024-05-17T15:23:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12849a8edc8d 2024-05-17T15:23:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12849a8edc8d 2024-05-17T15:23:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:09:27 2024 2024-05-17T15:23:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:09:27 2024 2024-05-17T15:23:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:09:27 2024 2024-05-17T15:23:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: d0b80867-2a3b-4947-ae2d-30cd6b4ea8cc 2024-05-17T15:23:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_27 journal status 2024-05-17T15:23:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_27 journal status' 2024-05-17T15:23:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_27 2024-05-17T15:23:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_27 2024-05-17T15:23:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_27 snapshots 2024-05-17T15:23:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_27 snapshots' 2024-05-17T15:23:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_27 2024-05-17T15:23:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 61 .mirror.non_primary.d0b80867-2a3b-4947-ae2d-30cd6b4ea8cc.6089583f-35a4-48b6-bf8a-35e7965a1dc8 128 MiB Fri May 17 15:09:28 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:61 copied) 2024-05-17T15:23:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_28 info' 2024-05-17T15:23:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_28 info 2024-05-17T15:23:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_28 2024-05-17T15:23:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_28': 2024-05-17T15:23:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12a36c6d2380 2024-05-17T15:23:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12a36c6d2380 2024-05-17T15:23:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:09:41 2024 2024-05-17T15:23:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:09:41 2024 2024-05-17T15:23:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:09:41 2024 2024-05-17T15:23:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 90f08d14-8ef0-4bff-9f0d-d804ed59aad1 2024-05-17T15:23:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:20.542 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_28 journal status 2024-05-17T15:23:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_28 journal status' 2024-05-17T15:23:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_28 2024-05-17T15:23:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_28 2024-05-17T15:23:20.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.588 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_28 snapshots 2024-05-17T15:23:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_28 snapshots' 2024-05-17T15:23:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_28 2024-05-17T15:23:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 62 .mirror.non_primary.90f08d14-8ef0-4bff-9f0d-d804ed59aad1.8fa97ec4-2472-4b8c-8cf2-47438775ad57 128 MiB Fri May 17 15:09:42 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:62 copied) 2024-05-17T15:23:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_29 info 2024-05-17T15:23:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_29 info' 2024-05-17T15:23:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_29 2024-05-17T15:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_29': 2024-05-17T15:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d36c132724 2024-05-17T15:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d36c132724 2024-05-17T15:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:09:47 2024 2024-05-17T15:23:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:09:47 2024 2024-05-17T15:23:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:09:47 2024 2024-05-17T15:23:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 59928e42-fdf9-4cdf-a1c9-1cabd73a05c5 2024-05-17T15:23:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_29 journal status 2024-05-17T15:23:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_29 journal status' 2024-05-17T15:23:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_29 2024-05-17T15:23:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_29 2024-05-17T15:23:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_29 snapshots 2024-05-17T15:23:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_29 snapshots' 2024-05-17T15:23:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_29 2024-05-17T15:23:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 63 .mirror.non_primary.59928e42-fdf9-4cdf-a1c9-1cabd73a05c5.d21789fd-d8bb-415f-b3fe-25cfd87ba090 128 MiB Fri May 17 15:09:48 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:63 copied) 2024-05-17T15:23:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_3 info 2024-05-17T15:23:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_3 info' 2024-05-17T15:23:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_3 2024-05-17T15:23:20.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_3': 2024-05-17T15:23:20.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 11358d942a71 2024-05-17T15:23:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.11358d942a71 2024-05-17T15:23:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:02 2024 2024-05-17T15:23:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:14:24 2024 2024-05-17T15:23:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:02 2024 2024-05-17T15:23:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 5191746b-1578-4957-a477-41bde526d1e0 2024-05-17T15:23:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.813 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_3 journal status 2024-05-17T15:23:20.813 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_3 journal status' 2024-05-17T15:23:20.813 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_3 2024-05-17T15:23:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_3 2024-05-17T15:23:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_3 snapshots 2024-05-17T15:23:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_3 snapshots' 2024-05-17T15:23:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_3 2024-05-17T15:23:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 89 snap 128 MiB Fri May 17 15:14:20 2024 user 2024-05-17T15:23:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 90 .mirror.non_primary.5191746b-1578-4957-a477-41bde526d1e0.09c45a96-3f7d-4ae6-ac5c-07664b8d09b0 128 MiB Fri May 17 15:14:21 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:90 copied) 2024-05-17T15:23:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_30 info 2024-05-17T15:23:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_30 info' 2024-05-17T15:23:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_30 2024-05-17T15:23:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_30': 2024-05-17T15:23:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1135ea3ef69e 2024-05-17T15:23:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1135ea3ef69e 2024-05-17T15:23:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:10:00 2024 2024-05-17T15:23:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:10:00 2024 2024-05-17T15:23:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:10:00 2024 2024-05-17T15:23:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 2031e3c2-db04-4d88-a9f5-2a8ddf1e9fce 2024-05-17T15:23:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_30 journal status 2024-05-17T15:23:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_30 journal status' 2024-05-17T15:23:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_30 2024-05-17T15:23:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_30 2024-05-17T15:23:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_30 snapshots 2024-05-17T15:23:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_30 snapshots' 2024-05-17T15:23:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_30 2024-05-17T15:23:21.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:21.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 64 .mirror.non_primary.2031e3c2-db04-4d88-a9f5-2a8ddf1e9fce.eb4537c2-f692-415c-9718-364f42bd3a64 128 MiB Fri May 17 15:10:01 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:64 copied) 2024-05-17T15:23:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_31 info 2024-05-17T15:23:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_31 info' 2024-05-17T15:23:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_31 2024-05-17T15:23:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_31': 2024-05-17T15:23:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12adc6efeed4 2024-05-17T15:23:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12adc6efeed4 2024-05-17T15:23:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:10:13 2024 2024-05-17T15:23:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:10:13 2024 2024-05-17T15:23:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:10:13 2024 2024-05-17T15:23:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 219fe1bd-503a-4377-b187-f83062f5b1ba 2024-05-17T15:23:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:21.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_31 journal status 2024-05-17T15:23:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_31 journal status' 2024-05-17T15:23:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_31 2024-05-17T15:23:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_31 2024-05-17T15:23:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_31 snapshots 2024-05-17T15:23:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_31 snapshots' 2024-05-17T15:23:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_31 2024-05-17T15:23:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 65 .mirror.non_primary.219fe1bd-503a-4377-b187-f83062f5b1ba.c2583847-e69a-4fbc-9396-c5a4d6f74a19 128 MiB Fri May 17 15:10:14 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:65 copied) 2024-05-17T15:23:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_32 info' 2024-05-17T15:23:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_32 info 2024-05-17T15:23:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_32 2024-05-17T15:23:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_32': 2024-05-17T15:23:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12adc2237663 2024-05-17T15:23:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12adc2237663 2024-05-17T15:23:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:10:28 2024 2024-05-17T15:23:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:10:28 2024 2024-05-17T15:23:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:10:28 2024 2024-05-17T15:23:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 0835fff5-1be2-449a-8ff6-85a5ed6b71f8 2024-05-17T15:23:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:21.217 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_32 journal status 2024-05-17T15:23:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_32 journal status' 2024-05-17T15:23:21.218 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_32 2024-05-17T15:23:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_32 2024-05-17T15:23:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_32 snapshots 2024-05-17T15:23:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_32 snapshots' 2024-05-17T15:23:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_32 2024-05-17T15:23:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 66 .mirror.non_primary.0835fff5-1be2-449a-8ff6-85a5ed6b71f8.83eef303-06dc-4bac-a162-402b506be118 128 MiB Fri May 17 15:10:29 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:66 copied) 2024-05-17T15:23:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_33 info 2024-05-17T15:23:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_33 info' 2024-05-17T15:23:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_33 2024-05-17T15:23:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_33': 2024-05-17T15:23:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d355afdfdb 2024-05-17T15:23:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d355afdfdb 2024-05-17T15:23:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:10:37 2024 2024-05-17T15:23:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:10:37 2024 2024-05-17T15:23:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:10:37 2024 2024-05-17T15:23:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 0218530c-9285-4e1b-ac61-baf8583aceb9 2024-05-17T15:23:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_33 journal status 2024-05-17T15:23:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_33 journal status' 2024-05-17T15:23:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_33 2024-05-17T15:23:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_33 2024-05-17T15:23:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_33 snapshots 2024-05-17T15:23:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_33 snapshots' 2024-05-17T15:23:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_33 2024-05-17T15:23:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:21.444 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 67 .mirror.non_primary.0218530c-9285-4e1b-ac61-baf8583aceb9.b67b5a52-c0a2-452d-94d3-073da5583ead 128 MiB Fri May 17 15:10:38 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:67 copied) 2024-05-17T15:23:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_34 info 2024-05-17T15:23:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_34 info' 2024-05-17T15:23:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_34 2024-05-17T15:23:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_34': 2024-05-17T15:23:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1284a94e4c87 2024-05-17T15:23:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1284a94e4c87 2024-05-17T15:23:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:10:44 2024 2024-05-17T15:23:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:10:44 2024 2024-05-17T15:23:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:10:44 2024 2024-05-17T15:23:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 32af895d-8b5f-4d18-99db-b2ad45a7ac69 2024-05-17T15:23:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_34 journal status 2024-05-17T15:23:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_34 journal status' 2024-05-17T15:23:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_34 2024-05-17T15:23:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_34 2024-05-17T15:23:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_34 snapshots 2024-05-17T15:23:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_34 snapshots' 2024-05-17T15:23:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_34 2024-05-17T15:23:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 68 .mirror.non_primary.32af895d-8b5f-4d18-99db-b2ad45a7ac69.0426ef98-ec08-4e10-8d5d-df8add8035b3 128 MiB Fri May 17 15:10:45 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:68 copied) 2024-05-17T15:23:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_35 info 2024-05-17T15:23:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_35 info' 2024-05-17T15:23:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_35 2024-05-17T15:23:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_35': 2024-05-17T15:23:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12a3a48a4c32 2024-05-17T15:23:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12a3a48a4c32 2024-05-17T15:23:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:10:55 2024 2024-05-17T15:23:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:10:55 2024 2024-05-17T15:23:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:10:55 2024 2024-05-17T15:23:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 8c639172-6810-4a20-a5ee-19b5a50ec6fd 2024-05-17T15:23:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:21.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_35 journal status 2024-05-17T15:23:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_35 journal status' 2024-05-17T15:23:21.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_35 2024-05-17T15:23:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_35 2024-05-17T15:23:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_35 snapshots 2024-05-17T15:23:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_35 snapshots' 2024-05-17T15:23:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_35 2024-05-17T15:23:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 69 .mirror.non_primary.8c639172-6810-4a20-a5ee-19b5a50ec6fd.cd255ac1-02db-4019-be55-76a569050796 128 MiB Fri May 17 15:10:56 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:69 copied) 2024-05-17T15:23:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_36 info' 2024-05-17T15:23:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_36 info 2024-05-17T15:23:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_36 2024-05-17T15:23:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_36': 2024-05-17T15:23:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12ad1f2c19f8 2024-05-17T15:23:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12ad1f2c19f8 2024-05-17T15:23:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:11:09 2024 2024-05-17T15:23:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:11:09 2024 2024-05-17T15:23:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:11:09 2024 2024-05-17T15:23:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: b8165b54-ecda-4ca3-8d0a-f0c1a85112c7 2024-05-17T15:23:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_36 journal status 2024-05-17T15:23:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_36 journal status' 2024-05-17T15:23:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_36 2024-05-17T15:23:21.805 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_36 2024-05-17T15:23:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_36 snapshots 2024-05-17T15:23:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_36 snapshots' 2024-05-17T15:23:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_36 2024-05-17T15:23:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 70 .mirror.non_primary.b8165b54-ecda-4ca3-8d0a-f0c1a85112c7.e7efa06b-0247-4390-b11d-4fa6c274d5c8 128 MiB Fri May 17 15:11:10 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:70 copied) 2024-05-17T15:23:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_37 info' 2024-05-17T15:23:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_37 info 2024-05-17T15:23:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_37 2024-05-17T15:23:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_37': 2024-05-17T15:23:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d3f22caa6a 2024-05-17T15:23:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d3f22caa6a 2024-05-17T15:23:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:11:24 2024 2024-05-17T15:23:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:11:24 2024 2024-05-17T15:23:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:11:24 2024 2024-05-17T15:23:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 1cbb263d-bbcf-4037-8f80-52c0b1dfd284 2024-05-17T15:23:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_37 journal status 2024-05-17T15:23:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_37 journal status' 2024-05-17T15:23:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_37 2024-05-17T15:23:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_37 2024-05-17T15:23:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_37 snapshots 2024-05-17T15:23:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_37 snapshots' 2024-05-17T15:23:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_37 2024-05-17T15:23:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 71 .mirror.non_primary.1cbb263d-bbcf-4037-8f80-52c0b1dfd284.fbb3e9f0-692d-4124-be6a-6177a6f76eea 128 MiB Fri May 17 15:11:25 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:71 copied) 2024-05-17T15:23:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_38 info 2024-05-17T15:23:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_38 info' 2024-05-17T15:23:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_38 2024-05-17T15:23:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_38': 2024-05-17T15:23:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1284e73f1092 2024-05-17T15:23:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1284e73f1092 2024-05-17T15:23:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:11:33 2024 2024-05-17T15:23:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:11:33 2024 2024-05-17T15:23:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:11:33 2024 2024-05-17T15:23:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: f990d936-a771-483b-8728-fd1503201136 2024-05-17T15:23:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_38 journal status 2024-05-17T15:23:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_38 journal status' 2024-05-17T15:23:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_38 2024-05-17T15:23:22.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_38 2024-05-17T15:23:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_38 snapshots 2024-05-17T15:23:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_38 snapshots' 2024-05-17T15:23:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_38 2024-05-17T15:23:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 72 .mirror.non_primary.f990d936-a771-483b-8728-fd1503201136.2943fc12-577f-48c8-9d5d-c8e1a4a82b11 128 MiB Fri May 17 15:11:34 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:72 copied) 2024-05-17T15:23:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_39 info 2024-05-17T15:23:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_39 info' 2024-05-17T15:23:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_39 2024-05-17T15:23:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_39': 2024-05-17T15:23:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12a3a111b8cb 2024-05-17T15:23:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12a3a111b8cb 2024-05-17T15:23:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:11:45 2024 2024-05-17T15:23:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:11:45 2024 2024-05-17T15:23:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:11:45 2024 2024-05-17T15:23:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: aae21142-bf9f-4d97-9ad7-8e4679ee037a 2024-05-17T15:23:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_39 journal status' 2024-05-17T15:23:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_39 journal status 2024-05-17T15:23:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_39 2024-05-17T15:23:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_39 2024-05-17T15:23:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_39 snapshots 2024-05-17T15:23:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_39 snapshots' 2024-05-17T15:23:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_39 2024-05-17T15:23:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 73 .mirror.non_primary.aae21142-bf9f-4d97-9ad7-8e4679ee037a.fbbfa4f4-edfc-48f1-a410-ce054c7ef8a3 128 MiB Fri May 17 15:11:46 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:73 copied) 2024-05-17T15:23:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_4 info 2024-05-17T15:23:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_4 info' 2024-05-17T15:23:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_4 2024-05-17T15:23:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_4': 2024-05-17T15:23:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 121df7b93334 2024-05-17T15:23:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.121df7b93334 2024-05-17T15:23:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:10 2024 2024-05-17T15:23:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:14:32 2024 2024-05-17T15:23:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:10 2024 2024-05-17T15:23:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 1fda3480-fa61-4b4b-93a6-2dd063ac50f4 2024-05-17T15:23:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_4 journal status 2024-05-17T15:23:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_4 journal status' 2024-05-17T15:23:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_4 2024-05-17T15:23:22.351 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_4 2024-05-17T15:23:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_4 snapshots' 2024-05-17T15:23:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_4 snapshots 2024-05-17T15:23:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_4 2024-05-17T15:23:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 91 snap 128 MiB Fri May 17 15:14:28 2024 user 2024-05-17T15:23:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 92 .mirror.non_primary.1fda3480-fa61-4b4b-93a6-2dd063ac50f4.e6d62e5e-350f-465d-8332-8d52ccc67d52 128 MiB Fri May 17 15:14:29 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:92 copied) 2024-05-17T15:23:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_40 info 2024-05-17T15:23:22.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_40 info' 2024-05-17T15:23:22.400 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_40 2024-05-17T15:23:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_40': 2024-05-17T15:23:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12ad2ccfa334 2024-05-17T15:23:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12ad2ccfa334 2024-05-17T15:23:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:11:51 2024 2024-05-17T15:23:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:11:51 2024 2024-05-17T15:23:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:11:51 2024 2024-05-17T15:23:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 4abdb521-afae-4334-ad13-70815420eba8 2024-05-17T15:23:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_40 journal status 2024-05-17T15:23:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_40 journal status' 2024-05-17T15:23:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_40 2024-05-17T15:23:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_40 2024-05-17T15:23:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_40 snapshots 2024-05-17T15:23:22.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_40 snapshots' 2024-05-17T15:23:22.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_40 2024-05-17T15:23:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 74 .mirror.non_primary.4abdb521-afae-4334-ad13-70815420eba8.a229ce1f-9aeb-4132-a886-2b5df5dd9ecd 128 MiB Fri May 17 15:11:52 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:74 copied) 2024-05-17T15:23:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_41 info 2024-05-17T15:23:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_41 info' 2024-05-17T15:23:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_41 2024-05-17T15:23:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_41': 2024-05-17T15:23:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d382634ca3 2024-05-17T15:23:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d382634ca3 2024-05-17T15:23:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:12:03 2024 2024-05-17T15:23:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:12:03 2024 2024-05-17T15:23:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:12:03 2024 2024-05-17T15:23:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 69d8440b-93d1-40e8-b675-4ab5105af9bd 2024-05-17T15:23:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:22.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.584 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_41 journal status 2024-05-17T15:23:22.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_41 journal status' 2024-05-17T15:23:22.585 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_41 2024-05-17T15:23:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_41 2024-05-17T15:23:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_41 snapshots 2024-05-17T15:23:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_41 snapshots' 2024-05-17T15:23:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_41 2024-05-17T15:23:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 75 .mirror.non_primary.69d8440b-93d1-40e8-b675-4ab5105af9bd.768dfc8d-0f01-4f04-83fb-750c6c744272 128 MiB Fri May 17 15:12:04 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:75 copied) 2024-05-17T15:23:22.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_42 info 2024-05-17T15:23:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_42 info' 2024-05-17T15:23:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_42 2024-05-17T15:23:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_42': 2024-05-17T15:23:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1284ad92c3a1 2024-05-17T15:23:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1284ad92c3a1 2024-05-17T15:23:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:12:11 2024 2024-05-17T15:23:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:12:11 2024 2024-05-17T15:23:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:12:11 2024 2024-05-17T15:23:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 46ae8559-1830-4c3d-adfc-c8f96345efad 2024-05-17T15:23:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_42 journal status 2024-05-17T15:23:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_42 journal status' 2024-05-17T15:23:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_42 2024-05-17T15:23:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_42 2024-05-17T15:23:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_42 snapshots 2024-05-17T15:23:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_42 snapshots' 2024-05-17T15:23:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_42 2024-05-17T15:23:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 76 .mirror.non_primary.46ae8559-1830-4c3d-adfc-c8f96345efad.36b3e2ed-a66c-4634-beed-8dd5cb4ecb3f 128 MiB Fri May 17 15:12:12 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:76 copied) 2024-05-17T15:23:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_43 info 2024-05-17T15:23:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_43 info' 2024-05-17T15:23:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_43 2024-05-17T15:23:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_43': 2024-05-17T15:23:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12a33ce1f7e6 2024-05-17T15:23:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12a33ce1f7e6 2024-05-17T15:23:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:12:18 2024 2024-05-17T15:23:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:12:18 2024 2024-05-17T15:23:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:12:18 2024 2024-05-17T15:23:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 414f7604-42d0-4537-bea3-bfcfbe66e210 2024-05-17T15:23:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_43 journal status 2024-05-17T15:23:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_43 journal status' 2024-05-17T15:23:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_43 2024-05-17T15:23:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_43 2024-05-17T15:23:22.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_43 snapshots 2024-05-17T15:23:22.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_43 snapshots' 2024-05-17T15:23:22.897 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_43 2024-05-17T15:23:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 77 .mirror.non_primary.414f7604-42d0-4537-bea3-bfcfbe66e210.14d129b1-169e-4b10-ad0d-8e354527d5a8 128 MiB Fri May 17 15:12:19 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:77 copied) 2024-05-17T15:23:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_44 info 2024-05-17T15:23:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_44 info' 2024-05-17T15:23:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_44 2024-05-17T15:23:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_44': 2024-05-17T15:23:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12ade767e69f 2024-05-17T15:23:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12ade767e69f 2024-05-17T15:23:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:12:24 2024 2024-05-17T15:23:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:12:24 2024 2024-05-17T15:23:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:12:24 2024 2024-05-17T15:23:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 42d7dd77-f5e1-4c19-a790-e2073f5b67ee 2024-05-17T15:23:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_44 journal status 2024-05-17T15:23:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_44 journal status' 2024-05-17T15:23:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_44 2024-05-17T15:23:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_44 2024-05-17T15:23:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:23.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_44 snapshots 2024-05-17T15:23:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_44 snapshots' 2024-05-17T15:23:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_44 2024-05-17T15:23:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 78 .mirror.non_primary.42d7dd77-f5e1-4c19-a790-e2073f5b67ee.0e51bee7-93ac-40e2-8c22-ded0cb89eaed 128 MiB Fri May 17 15:12:25 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:78 copied) 2024-05-17T15:23:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_45 info 2024-05-17T15:23:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_45 info' 2024-05-17T15:23:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_45 2024-05-17T15:23:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_45': 2024-05-17T15:23:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d3c5666afd 2024-05-17T15:23:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d3c5666afd 2024-05-17T15:23:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:12:31 2024 2024-05-17T15:23:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:12:31 2024 2024-05-17T15:23:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:12:31 2024 2024-05-17T15:23:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 729f0874-9d81-4812-83cf-ae9935d33036 2024-05-17T15:23:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_45 journal status 2024-05-17T15:23:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_45 journal status' 2024-05-17T15:23:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_45 2024-05-17T15:23:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_45 2024-05-17T15:23:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_45 snapshots 2024-05-17T15:23:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_45 snapshots' 2024-05-17T15:23:23.168 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_45 2024-05-17T15:23:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 79 .mirror.non_primary.729f0874-9d81-4812-83cf-ae9935d33036.aeaf4373-086e-4c06-a097-b36c11555b30 128 MiB Fri May 17 15:12:32 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:79 copied) 2024-05-17T15:23:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_46 info' 2024-05-17T15:23:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_46 info 2024-05-17T15:23:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_46 2024-05-17T15:23:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_46': 2024-05-17T15:23:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12845014c61a 2024-05-17T15:23:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12845014c61a 2024-05-17T15:23:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:12:46 2024 2024-05-17T15:23:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:12:46 2024 2024-05-17T15:23:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:12:46 2024 2024-05-17T15:23:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 36a3913d-378a-412a-ad5b-9c269860173e 2024-05-17T15:23:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:23.256 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_46 journal status 2024-05-17T15:23:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_46 journal status' 2024-05-17T15:23:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_46 2024-05-17T15:23:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_46 2024-05-17T15:23:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:23.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:23.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_46 snapshots 2024-05-17T15:23:23.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_46 snapshots' 2024-05-17T15:23:23.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_46 2024-05-17T15:23:23.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 80 .mirror.non_primary.36a3913d-378a-412a-ad5b-9c269860173e.8ab22aa8-6d3b-47ae-a2f5-9a9a53612984 128 MiB Fri May 17 15:12:47 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:80 copied) 2024-05-17T15:23:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_47 info 2024-05-17T15:23:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_47 info' 2024-05-17T15:23:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_47 2024-05-17T15:23:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_47': 2024-05-17T15:23:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12a39a9f7909 2024-05-17T15:23:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12a39a9f7909 2024-05-17T15:23:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:12:52 2024 2024-05-17T15:23:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:12:52 2024 2024-05-17T15:23:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:12:52 2024 2024-05-17T15:23:23.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:23.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:23.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 84e9e05b-5031-4201-93ab-915cf8a3c889 2024-05-17T15:23:23.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:23.596 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:23.596 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_47 journal status 2024-05-17T15:23:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_47 journal status' 2024-05-17T15:23:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_47 2024-05-17T15:23:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_47 2024-05-17T15:23:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_47 snapshots' 2024-05-17T15:23:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_47 snapshots 2024-05-17T15:23:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_47 2024-05-17T15:23:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 81 .mirror.non_primary.84e9e05b-5031-4201-93ab-915cf8a3c889.77ba4699-5183-44ec-bd89-9a6e5e75f7bd 128 MiB Fri May 17 15:12:53 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:81 copied) 2024-05-17T15:23:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_48 info 2024-05-17T15:23:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_48 info' 2024-05-17T15:23:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_48 2024-05-17T15:23:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_48': 2024-05-17T15:23:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12ad5935de3e 2024-05-17T15:23:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12ad5935de3e 2024-05-17T15:23:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:13:02 2024 2024-05-17T15:23:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:13:02 2024 2024-05-17T15:23:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:13:02 2024 2024-05-17T15:23:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 84523fc1-2802-4868-9b68-13d10e9be7e2 2024-05-17T15:23:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:23.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_48 journal status 2024-05-17T15:23:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_48 journal status' 2024-05-17T15:23:23.733 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_48 2024-05-17T15:23:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_48 2024-05-17T15:23:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_48 snapshots 2024-05-17T15:23:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_48 snapshots' 2024-05-17T15:23:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_48 2024-05-17T15:23:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:23.820 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 82 .mirror.non_primary.84523fc1-2802-4868-9b68-13d10e9be7e2.0f389266-89b0-4228-a10e-afe6210f8fe5 128 MiB Fri May 17 15:13:03 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:82 copied) 2024-05-17T15:23:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_49 info 2024-05-17T15:23:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_49 info' 2024-05-17T15:23:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_49 2024-05-17T15:23:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_49': 2024-05-17T15:23:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d39ab90e04 2024-05-17T15:23:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d39ab90e04 2024-05-17T15:23:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:13:14 2024 2024-05-17T15:23:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:13:14 2024 2024-05-17T15:23:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:13:14 2024 2024-05-17T15:23:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 6c86503a-19c3-470e-81c4-44311421ecfd 2024-05-17T15:23:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_49 journal status 2024-05-17T15:23:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_49 journal status' 2024-05-17T15:23:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_49 2024-05-17T15:23:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_49 2024-05-17T15:23:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_49 snapshots 2024-05-17T15:23:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_49 snapshots' 2024-05-17T15:23:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_49 2024-05-17T15:23:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 83 .mirror.non_primary.6c86503a-19c3-470e-81c4-44311421ecfd.3fec2997-b468-4798-bdb0-c37210c8b8f2 128 MiB Fri May 17 15:13:15 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:83 copied) 2024-05-17T15:23:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_5 info 2024-05-17T15:23:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_5 info' 2024-05-17T15:23:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_5 2024-05-17T15:23:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_5': 2024-05-17T15:23:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10c8713e805a 2024-05-17T15:23:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10c8713e805a 2024-05-17T15:23:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:17 2024 2024-05-17T15:23:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:14:41 2024 2024-05-17T15:23:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:17 2024 2024-05-17T15:23:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 82562a71-d364-4946-b5e9-67440f1071c4 2024-05-17T15:23:24.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_5 journal status 2024-05-17T15:23:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_5 journal status' 2024-05-17T15:23:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_5 2024-05-17T15:23:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_5 2024-05-17T15:23:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_5 snapshots 2024-05-17T15:23:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_5 snapshots' 2024-05-17T15:23:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_5 2024-05-17T15:23:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 93 snap 128 MiB Fri May 17 15:14:37 2024 user 2024-05-17T15:23:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 94 .mirror.non_primary.82562a71-d364-4946-b5e9-67440f1071c4.6a9134e3-e8ad-4f1b-b5c7-00c2d62f44b0 128 MiB Fri May 17 15:14:38 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:94 copied) 2024-05-17T15:23:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_50 info 2024-05-17T15:23:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_50 info' 2024-05-17T15:23:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_50 2024-05-17T15:23:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_50': 2024-05-17T15:23:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 1 2024-05-17T15:23:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 1284f40a8353 2024-05-17T15:23:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.1284f40a8353 2024-05-17T15:23:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:13:24 2024 2024-05-17T15:23:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:13:24 2024 2024-05-17T15:23:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:13:24 2024 2024-05-17T15:23:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 9ca9b16d-6fb0-46d4-9998-208fa4980f4f 2024-05-17T15:23:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:24.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.139 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_50 journal status 2024-05-17T15:23:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_50 journal status' 2024-05-17T15:23:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_50 2024-05-17T15:23:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_50 2024-05-17T15:23:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_50 snapshots 2024-05-17T15:23:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_50 snapshots' 2024-05-17T15:23:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_50 2024-05-17T15:23:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 84 .mirror.non_primary.9ca9b16d-6fb0-46d4-9998-208fa4980f4f.5684cf0e-45c7-47c6-ac99-5eb78a49ca6d 128 MiB Fri May 17 15:13:25 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:84 copied) 2024-05-17T15:23:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_6 info 2024-05-17T15:23:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_6 info' 2024-05-17T15:23:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_6 2024-05-17T15:23:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_6': 2024-05-17T15:23:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 12844609881d 2024-05-17T15:23:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.12844609881d 2024-05-17T15:23:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:27 2024 2024-05-17T15:23:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:14:49 2024 2024-05-17T15:23:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:27 2024 2024-05-17T15:23:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: b40480e1-3aef-46bb-a86f-05a1ee0d2d26 2024-05-17T15:23:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_6 journal status 2024-05-17T15:23:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_6 journal status' 2024-05-17T15:23:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_6 2024-05-17T15:23:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_6 2024-05-17T15:23:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_6 snapshots 2024-05-17T15:23:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_6 snapshots' 2024-05-17T15:23:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_6 2024-05-17T15:23:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 95 snap 128 MiB Fri May 17 15:14:45 2024 user 2024-05-17T15:23:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 96 .mirror.non_primary.b40480e1-3aef-46bb-a86f-05a1ee0d2d26.2b94da91-32ef-459f-8cca-29dc70bcc06b 128 MiB Fri May 17 15:14:46 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:96 copied) 2024-05-17T15:23:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_7 info 2024-05-17T15:23:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_7 info' 2024-05-17T15:23:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_7 2024-05-17T15:23:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_7': 2024-05-17T15:23:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d332701993 2024-05-17T15:23:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d332701993 2024-05-17T15:23:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:35 2024 2024-05-17T15:23:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:14:57 2024 2024-05-17T15:23:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:35 2024 2024-05-17T15:23:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 6e4b54d1-2cee-4ccb-aef6-8e1d0ba0e670 2024-05-17T15:23:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_7 journal status 2024-05-17T15:23:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_7 journal status' 2024-05-17T15:23:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_7 2024-05-17T15:23:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_7 2024-05-17T15:23:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_7 snapshots 2024-05-17T15:23:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_7 snapshots' 2024-05-17T15:23:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_7 2024-05-17T15:23:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 97 snap 128 MiB Fri May 17 15:14:53 2024 user 2024-05-17T15:23:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 98 .mirror.non_primary.6e4b54d1-2cee-4ccb-aef6-8e1d0ba0e670.3ed5f7ab-5504-43d8-beda-380cfa2ef986 128 MiB Fri May 17 15:14:54 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:98 copied) 2024-05-17T15:23:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_8 info' 2024-05-17T15:23:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_8 info 2024-05-17T15:23:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_8 2024-05-17T15:23:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_8': 2024-05-17T15:23:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 11351efa57ba 2024-05-17T15:23:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.11351efa57ba 2024-05-17T15:23:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:50 2024 2024-05-17T15:23:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:15:05 2024 2024-05-17T15:23:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:50 2024 2024-05-17T15:23:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 75a0868a-edaf-438e-a7e1-fedf558dccfa 2024-05-17T15:23:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_8 journal status 2024-05-17T15:23:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_8 journal status' 2024-05-17T15:23:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_8 2024-05-17T15:23:24.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_8 2024-05-17T15:23:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_8 snapshots 2024-05-17T15:23:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_8 snapshots' 2024-05-17T15:23:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_8 2024-05-17T15:23:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 99 snap 128 MiB Fri May 17 15:15:01 2024 user 2024-05-17T15:23:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 100 .mirror.non_primary.75a0868a-edaf-438e-a7e1-fedf558dccfa.84c28084-da26-448e-be24-4082ea9038af 128 MiB Fri May 17 15:15:02 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:100 copied) 2024-05-17T15:23:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_9 info 2024-05-17T15:23:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image in `rbd --cluster ${cluster} -p ${image_pool} --namespace "${image_ns}" ls 2>/dev/null` 2024-05-17T15:23:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_9 info' 2024-05-17T15:23:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' info image_9 2024-05-17T15:23:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:rbd image 'image_9': 2024-05-17T15:23:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: size 128 MiB in 32 objects 2024-05-17T15:23:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: order 22 (4 MiB objects) 2024-05-17T15:23:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: snapshot_count: 2 2024-05-17T15:23:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: id: 10d349c640c3 2024-05-17T15:23:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: block_name_prefix: rbd_data.10d349c640c3 2024-05-17T15:23:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: format: 2 2024-05-17T15:23:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-05-17T15:23:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: op_features: 2024-05-17T15:23:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: flags: 2024-05-17T15:23:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: create_timestamp: Fri May 17 15:05:59 2024 2024-05-17T15:23:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: access_timestamp: Fri May 17 15:15:13 2024 2024-05-17T15:23:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: modify_timestamp: Fri May 17 15:05:59 2024 2024-05-17T15:23:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring state: enabled 2024-05-17T15:23:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring mode: snapshot 2024-05-17T15:23:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring global id: 6c3aab86-8ad4-4a10-8f48-3d66ec7f5b30 2024-05-17T15:23:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: mirroring primary: false 2024-05-17T15:23:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_9 journal status 2024-05-17T15:23:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_9 journal status' 2024-05-17T15:23:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' journal status --image image_9 2024-05-17T15:23:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: journaling is not enabled for image image_9 2024-05-17T15:23:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image image_9 snapshots 2024-05-17T15:23:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'image image_9 snapshots' 2024-05-17T15:23:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' snap ls --all image_9 2024-05-17T15:23:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2024-05-17T15:23:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 101 snap 128 MiB Fri May 17 15:15:09 2024 user 2024-05-17T15:23:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 102 .mirror.non_primary.6c3aab86-8ad4-4a10-8f48-3d66ec7f5b30.39f8b34c-a39d-4038-8a8c-d2ce0ed26973 128 MiB Fri May 17 15:15:10 2024 mirror (non-primary peer_uuids:[] 7e816122-c11e-4a54-9bab-7372f406213b:102 copied) 2024-05-17T15:23:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror rbd_mirroring omap vals 2024-05-17T15:23:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror rbd_mirroring omap vals' 2024-05-17T15:23:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirroring 2024-05-17T15:23:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_0218530c-9285-4e1b-ac61-baf8583aceb9 2024-05-17T15:23:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 30 64 33 35 35 61 66 64 66 64 62 |....10d355afdfdb| 2024-05-17T15:23:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_0835fff5-1be2-449a-8ff6-85a5ed6b71f8 2024-05-17T15:23:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 61 64 63 32 32 33 37 36 36 33 |....12adc2237663| 2024-05-17T15:23:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_0e1e48a7-bf0e-42d2-9d09-07eeeeaaf56e 2024-05-17T15:23:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 31 64 64 63 38 34 62 34 65 39 |....121ddc84b4e9| 2024-05-17T15:23:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_1cbb263d-bbcf-4037-8f80-52c0b1dfd284 2024-05-17T15:23:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 30 64 33 66 32 32 63 61 61 36 61 |....10d3f22caa6a| 2024-05-17T15:23:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_1fda3480-fa61-4b4b-93a6-2dd063ac50f4 2024-05-17T15:23:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 31 64 66 37 62 39 33 33 33 34 |....121df7b93334| 2024-05-17T15:23:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_2031e3c2-db04-4d88-a9f5-2a8ddf1e9fce 2024-05-17T15:23:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 31 33 35 65 61 33 65 66 36 39 65 |....1135ea3ef69e| 2024-05-17T15:23:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_219fe1bd-503a-4377-b187-f83062f5b1ba 2024-05-17T15:23:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 61 64 63 36 65 66 65 65 64 34 |....12adc6efeed4| 2024-05-17T15:23:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_2968dbed-dc66-4e82-be5f-bc6e3322a6c1 2024-05-17T15:23:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 38 34 63 31 36 64 61 35 31 36 |....1284c16da516| 2024-05-17T15:23:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_32af895d-8b5f-4d18-99db-b2ad45a7ac69 2024-05-17T15:23:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 38 34 61 39 34 65 34 63 38 37 |....1284a94e4c87| 2024-05-17T15:23:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_36a3913d-378a-412a-ad5b-9c269860173e 2024-05-17T15:23:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 38 34 35 30 31 34 63 36 31 61 |....12845014c61a| 2024-05-17T15:23:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_3df98478-a489-4a33-b01c-132fa8241534 2024-05-17T15:23:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 38 34 37 39 62 37 30 32 38 62 |....128479b7028b| 2024-05-17T15:23:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_414f7604-42d0-4537-bea3-bfcfbe66e210 2024-05-17T15:23:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 61 33 33 63 65 31 66 37 65 36 |....12a33ce1f7e6| 2024-05-17T15:23:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_42d7dd77-f5e1-4c19-a790-e2073f5b67ee 2024-05-17T15:23:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 61 64 65 37 36 37 65 36 39 66 |....12ade767e69f| 2024-05-17T15:23:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_46ae8559-1830-4c3d-adfc-c8f96345efad 2024-05-17T15:23:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 38 34 61 64 39 32 63 33 61 31 |....1284ad92c3a1| 2024-05-17T15:23:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_4abdb521-afae-4334-ad13-70815420eba8 2024-05-17T15:23:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 61 64 32 63 63 66 61 33 33 34 |....12ad2ccfa334| 2024-05-17T15:23:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_5191746b-1578-4957-a477-41bde526d1e0 2024-05-17T15:23:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 31 33 35 38 64 39 34 32 61 37 31 |....11358d942a71| 2024-05-17T15:23:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_59928e42-fdf9-4cdf-a1c9-1cabd73a05c5 2024-05-17T15:23:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 30 64 33 36 63 31 33 32 37 32 34 |....10d36c132724| 2024-05-17T15:23:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_5f4f313b-d9df-420f-8c92-2d9e481184bc 2024-05-17T15:23:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 61 33 62 61 35 61 61 36 34 35 |....12a3ba5aa645| 2024-05-17T15:23:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_69d8440b-93d1-40e8-b675-4ab5105af9bd 2024-05-17T15:23:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 30 64 33 38 32 36 33 34 63 61 33 |....10d382634ca3| 2024-05-17T15:23:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_6c3aab86-8ad4-4a10-8f48-3d66ec7f5b30 2024-05-17T15:23:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 30 64 33 34 39 63 36 34 30 63 33 |....10d349c640c3| 2024-05-17T15:23:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_6c86503a-19c3-470e-81c4-44311421ecfd 2024-05-17T15:23:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 30 64 33 39 61 62 39 30 65 30 34 |....10d39ab90e04| 2024-05-17T15:23:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_6e4b54d1-2cee-4ccb-aef6-8e1d0ba0e670 2024-05-17T15:23:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 30 64 33 33 32 37 30 31 39 39 33 |....10d332701993| 2024-05-17T15:23:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_729f0874-9d81-4812-83cf-ae9935d33036 2024-05-17T15:23:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 30 64 33 63 35 36 36 36 61 66 64 |....10d3c5666afd| 2024-05-17T15:23:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_75a0868a-edaf-438e-a7e1-fedf558dccfa 2024-05-17T15:23:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 31 33 35 31 65 66 61 35 37 62 61 |....11351efa57ba| 2024-05-17T15:23:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_76d62c45-9c33-4b38-aeb9-d12dc0c2247a 2024-05-17T15:23:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 30 64 33 64 35 35 39 30 39 39 35 |....10d3d5590995| 2024-05-17T15:23:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_7b475e11-a35f-4917-9f29-282df00e1098 2024-05-17T15:23:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 31 33 35 39 34 31 32 36 30 36 31 |....113594126061| 2024-05-17T15:23:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_7fede8a9-d664-48e9-b43a-740b49fc17be 2024-05-17T15:23:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 30 64 33 61 64 65 31 31 65 36 65 |....10d3ade11e6e| 2024-05-17T15:23:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_82562a71-d364-4946-b5e9-67440f1071c4 2024-05-17T15:23:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 30 63 38 37 31 33 65 38 30 35 61 |....10c8713e805a| 2024-05-17T15:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_84523fc1-2802-4868-9b68-13d10e9be7e2 2024-05-17T15:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 61 64 35 39 33 35 64 65 33 65 |....12ad5935de3e| 2024-05-17T15:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_84e9e05b-5031-4201-93ab-915cf8a3c889 2024-05-17T15:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 61 33 39 61 39 66 37 39 30 39 |....12a39a9f7909| 2024-05-17T15:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_8c639172-6810-4a20-a5ee-19b5a50ec6fd 2024-05-17T15:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 61 33 61 34 38 61 34 63 33 32 |....12a3a48a4c32| 2024-05-17T15:23:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_8d44a27c-5bdd-4a78-9b8a-9a4d22d34367 2024-05-17T15:23:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 30 64 33 62 36 64 37 36 64 65 32 |....10d3b6d76de2| 2024-05-17T15:23:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_90f08d14-8ef0-4bff-9f0d-d804ed59aad1 2024-05-17T15:23:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 61 33 36 63 36 64 32 33 38 30 |....12a36c6d2380| 2024-05-17T15:23:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_95950a54-837a-461f-aafc-36eac3b2a17a 2024-05-17T15:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 31 33 35 63 30 32 65 38 64 66 33 |....1135c02e8df3| 2024-05-17T15:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_99995afa-10de-4385-a379-455ec2be8a34 2024-05-17T15:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 39 33 39 33 31 30 62 38 65 38 |....12939310b8e8| 2024-05-17T15:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_9ca9b16d-6fb0-46d4-9998-208fa4980f4f 2024-05-17T15:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 38 34 66 34 30 61 38 33 35 33 |....1284f40a8353| 2024-05-17T15:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_9cc94201-e324-4319-a646-d571d0e3df05 2024-05-17T15:23:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 39 33 37 37 39 39 34 35 62 63 |....1293779945bc| 2024-05-17T15:23:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_9d50b403-8aa5-4da6-8e72-4a3f61c9358b 2024-05-17T15:23:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 31 33 35 36 62 30 36 34 31 31 66 |....11356b06411f| 2024-05-17T15:23:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_9e41e805-f938-4777-a0dd-ce32b1acfcf1 2024-05-17T15:23:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (15 bytes) : 2024-05-17T15:23:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0b 00 00 00 31 31 33 35 62 37 65 62 32 62 62 |....1135b7eb2bb| 2024-05-17T15:23:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000000f 2024-05-17T15:23:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_aae21142-bf9f-4d97-9ad7-8e4679ee037a 2024-05-17T15:23:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 61 33 61 31 31 31 62 38 63 62 |....12a3a111b8cb| 2024-05-17T15:23:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_b40480e1-3aef-46bb-a86f-05a1ee0d2d26 2024-05-17T15:23:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 38 34 34 36 30 39 38 38 31 64 |....12844609881d| 2024-05-17T15:23:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_b8165b54-ecda-4ca3-8d0a-f0c1a85112c7 2024-05-17T15:23:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 61 64 31 66 32 63 31 39 66 38 |....12ad1f2c19f8| 2024-05-17T15:23:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_bef15777-4cd7-47fb-8829-12dabc805b90 2024-05-17T15:23:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 30 64 33 31 39 37 34 64 35 64 61 |....10d31974d5da| 2024-05-17T15:23:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_c50f08e5-ef2e-4cc9-afd7-fb18dd3f03dc 2024-05-17T15:23:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 38 34 61 32 36 31 32 62 62 34 |....1284a2612bb4| 2024-05-17T15:23:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_ceaa7752-4708-4453-aad7-8c9fe4b5ad62 2024-05-17T15:23:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 30 63 38 66 30 65 38 64 33 37 37 |....10c8f0e8d377| 2024-05-17T15:23:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_d0b80867-2a3b-4947-ae2d-30cd6b4ea8cc 2024-05-17T15:23:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 38 34 39 61 38 65 64 63 38 64 |....12849a8edc8d| 2024-05-17T15:23:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_d95d4f33-deca-446d-88ca-2fd4ef56f554 2024-05-17T15:23:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 61 33 61 62 33 61 36 65 31 61 |....12a3ab3a6e1a| 2024-05-17T15:23:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_e3330572-d02a-4104-a14f-d83973929a2c 2024-05-17T15:23:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 31 33 35 63 34 63 30 33 62 32 33 |....1135c4c03b23| 2024-05-17T15:23:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_f990d936-a771-483b-8728-fd1503201136 2024-05-17T15:23:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 38 34 65 37 33 66 31 30 39 32 |....1284e73f1092| 2024-05-17T15:23:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:global_fb0ce489-3f67-44e1-971c-03b1005074a7 2024-05-17T15:23:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (16 bytes) : 2024-05-17T15:23:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 0c 00 00 00 31 32 39 33 64 63 34 64 35 31 39 36 |....1293dc4d5196| 2024-05-17T15:23:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 2024-05-17T15:23:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_10c8713e805a 2024-05-17T15:23:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 32 35 36 32 61 |..*...$...82562a| 2024-05-17T15:23:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 37 31 2d 64 33 36 34 2d 34 39 34 36 2d 62 35 65 |71-d364-4946-b5e| 2024-05-17T15:23:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 39 2d 36 37 34 34 30 66 31 30 37 31 63 34 01 01 |9-67440f1071c4..| 2024-05-17T15:23:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_10c8f0e8d377 2024-05-17T15:23:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 65 61 61 37 37 |..*...$...ceaa77| 2024-05-17T15:23:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 35 32 2d 34 37 30 38 2d 34 34 35 33 2d 61 61 64 |52-4708-4453-aad| 2024-05-17T15:23:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 37 2d 38 63 39 66 65 34 62 35 61 64 36 32 01 01 |7-8c9fe4b5ad62..| 2024-05-17T15:23:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_10d31974d5da 2024-05-17T15:23:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 65 66 31 35 37 |..*...$...bef157| 2024-05-17T15:23:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 37 37 2d 34 63 64 37 2d 34 37 66 62 2d 38 38 32 |77-4cd7-47fb-882| 2024-05-17T15:23:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 39 2d 31 32 64 61 62 63 38 30 35 62 39 30 01 01 |9-12dabc805b90..| 2024-05-17T15:23:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_10d332701993 2024-05-17T15:23:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 65 34 62 35 34 |..*...$...6e4b54| 2024-05-17T15:23:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 64 31 2d 32 63 65 65 2d 34 63 63 62 2d 61 65 66 |d1-2cee-4ccb-aef| 2024-05-17T15:23:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 36 2d 38 65 31 64 30 62 61 30 65 36 37 30 01 01 |6-8e1d0ba0e670..| 2024-05-17T15:23:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_10d349c640c3 2024-05-17T15:23:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 63 33 61 61 62 |..*...$...6c3aab| 2024-05-17T15:23:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 38 36 2d 38 61 64 34 2d 34 61 31 30 2d 38 66 34 |86-8ad4-4a10-8f4| 2024-05-17T15:23:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 38 2d 33 64 36 36 65 63 37 66 35 62 33 30 01 01 |8-3d66ec7f5b30..| 2024-05-17T15:23:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_10d355afdfdb 2024-05-17T15:23:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 30 32 31 38 35 33 |..*...$...021853| 2024-05-17T15:23:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 30 63 2d 39 32 38 35 2d 34 65 31 62 2d 61 63 36 |0c-9285-4e1b-ac6| 2024-05-17T15:23:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 31 2d 62 61 66 38 35 38 33 61 63 65 62 39 01 01 |1-baf8583aceb9..| 2024-05-17T15:23:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_10d36c132724 2024-05-17T15:23:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 39 39 32 38 65 |..*...$...59928e| 2024-05-17T15:23:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 34 32 2d 66 64 66 39 2d 34 63 64 66 2d 61 31 63 |42-fdf9-4cdf-a1c| 2024-05-17T15:23:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 39 2d 31 63 61 62 64 37 33 61 30 35 63 35 01 01 |9-1cabd73a05c5..| 2024-05-17T15:23:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_10d382634ca3 2024-05-17T15:23:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 39 64 38 34 34 |..*...$...69d844| 2024-05-17T15:23:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 30 62 2d 39 33 64 31 2d 34 30 65 38 2d 62 36 37 |0b-93d1-40e8-b67| 2024-05-17T15:23:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 35 2d 34 61 62 35 31 30 35 61 66 39 62 64 01 01 |5-4ab5105af9bd..| 2024-05-17T15:23:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_10d39ab90e04 2024-05-17T15:23:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 63 38 36 35 30 |..*...$...6c8650| 2024-05-17T15:23:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 33 61 2d 31 39 63 33 2d 34 37 30 65 2d 38 31 63 |3a-19c3-470e-81c| 2024-05-17T15:23:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 34 2d 34 34 33 31 31 34 32 31 65 63 66 64 01 01 |4-44311421ecfd..| 2024-05-17T15:23:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_10d3ade11e6e 2024-05-17T15:23:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 37 66 65 64 65 38 |..*...$...7fede8| 2024-05-17T15:23:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 61 39 2d 64 36 36 34 2d 34 38 65 39 2d 62 34 33 |a9-d664-48e9-b43| 2024-05-17T15:23:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 61 2d 37 34 30 62 34 39 66 63 31 37 62 65 01 01 |a-740b49fc17be..| 2024-05-17T15:23:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_10d3b6d76de2 2024-05-17T15:23:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 64 34 34 61 32 |..*...$...8d44a2| 2024-05-17T15:23:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 37 63 2d 35 62 64 64 2d 34 61 37 38 2d 39 62 38 |7c-5bdd-4a78-9b8| 2024-05-17T15:23:24.930 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 61 2d 39 61 34 64 32 32 64 33 34 33 36 37 01 01 |a-9a4d22d34367..| 2024-05-17T15:23:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_10d3c5666afd 2024-05-17T15:23:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 37 32 39 66 30 38 |..*...$...729f08| 2024-05-17T15:23:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 37 34 2d 39 64 38 31 2d 34 38 31 32 2d 38 33 63 |74-9d81-4812-83c| 2024-05-17T15:23:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 66 2d 61 65 39 39 33 35 64 33 33 30 33 36 01 01 |f-ae9935d33036..| 2024-05-17T15:23:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_10d3d5590995 2024-05-17T15:23:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 37 36 64 36 32 63 |..*...$...76d62c| 2024-05-17T15:23:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 34 35 2d 39 63 33 33 2d 34 62 33 38 2d 61 65 62 |45-9c33-4b38-aeb| 2024-05-17T15:23:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 39 2d 64 31 32 64 63 30 63 32 32 34 37 61 01 01 |9-d12dc0c2247a..| 2024-05-17T15:23:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_10d3f22caa6a 2024-05-17T15:23:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 63 62 62 32 36 |..*...$...1cbb26| 2024-05-17T15:23:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 33 64 2d 62 62 63 66 2d 34 30 33 37 2d 38 66 38 |3d-bbcf-4037-8f8| 2024-05-17T15:23:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 30 2d 35 32 63 30 62 31 64 66 64 32 38 34 01 01 |0-52c0b1dfd284..| 2024-05-17T15:23:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_11351efa57ba 2024-05-17T15:23:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 37 35 61 30 38 36 |..*...$...75a086| 2024-05-17T15:23:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 38 61 2d 65 64 61 66 2d 34 33 38 65 2d 61 37 65 |8a-edaf-438e-a7e| 2024-05-17T15:23:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 31 2d 66 65 64 66 35 35 38 64 63 63 66 61 01 01 |1-fedf558dccfa..| 2024-05-17T15:23:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_11356b06411f 2024-05-17T15:23:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 64 35 30 62 34 |..*...$...9d50b4| 2024-05-17T15:23:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 30 33 2d 38 61 61 35 2d 34 64 61 36 2d 38 65 37 |03-8aa5-4da6-8e7| 2024-05-17T15:23:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 32 2d 34 61 33 66 36 31 63 39 33 35 38 62 01 01 |2-4a3f61c9358b..| 2024-05-17T15:23:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_11358d942a71 2024-05-17T15:23:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 31 39 31 37 34 |..*...$...519174| 2024-05-17T15:23:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 36 62 2d 31 35 37 38 2d 34 39 35 37 2d 61 34 37 |6b-1578-4957-a47| 2024-05-17T15:23:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 37 2d 34 31 62 64 65 35 32 36 64 31 65 30 01 01 |7-41bde526d1e0..| 2024-05-17T15:23:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_113594126061 2024-05-17T15:23:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 37 62 34 37 35 65 |..*...$...7b475e| 2024-05-17T15:23:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 31 31 2d 61 33 35 66 2d 34 39 31 37 2d 39 66 32 |11-a35f-4917-9f2| 2024-05-17T15:23:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 39 2d 32 38 32 64 66 30 30 65 31 30 39 38 01 01 |9-282df00e1098..| 2024-05-17T15:23:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_1135b7eb2bb 2024-05-17T15:23:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 65 34 31 65 38 |..*...$...9e41e8| 2024-05-17T15:23:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 30 35 2d 66 39 33 38 2d 34 37 37 37 2d 61 30 64 |05-f938-4777-a0d| 2024-05-17T15:23:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 64 2d 63 65 33 32 62 31 61 63 66 63 66 31 01 01 |d-ce32b1acfcf1..| 2024-05-17T15:23:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_1135c02e8df3 2024-05-17T15:23:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 35 39 35 30 61 |..*...$...95950a| 2024-05-17T15:23:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 35 34 2d 38 33 37 61 2d 34 36 31 66 2d 61 61 66 |54-837a-461f-aaf| 2024-05-17T15:23:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 63 2d 33 36 65 61 63 33 62 32 61 31 37 61 01 01 |c-36eac3b2a17a..| 2024-05-17T15:23:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_1135c4c03b23 2024-05-17T15:23:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 33 33 33 30 35 |..*...$...e33305| 2024-05-17T15:23:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 37 32 2d 64 30 32 61 2d 34 31 30 34 2d 61 31 34 |72-d02a-4104-a14| 2024-05-17T15:23:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 66 2d 64 38 33 39 37 33 39 32 39 61 32 63 01 01 |f-d83973929a2c..| 2024-05-17T15:23:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_1135ea3ef69e 2024-05-17T15:23:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 32 30 33 31 65 33 |..*...$...2031e3| 2024-05-17T15:23:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 63 32 2d 64 62 30 34 2d 34 64 38 38 2d 61 39 66 |c2-db04-4d88-a9f| 2024-05-17T15:23:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 35 2d 32 61 38 64 64 66 31 65 39 66 63 65 01 01 |5-2a8ddf1e9fce..| 2024-05-17T15:23:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_121ddc84b4e9 2024-05-17T15:23:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 30 65 31 65 34 38 |..*...$...0e1e48| 2024-05-17T15:23:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 61 37 2d 62 66 30 65 2d 34 32 64 32 2d 39 64 30 |a7-bf0e-42d2-9d0| 2024-05-17T15:23:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 39 2d 30 37 65 65 65 65 61 61 66 35 36 65 01 01 |9-07eeeeaaf56e..| 2024-05-17T15:23:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_121df7b93334 2024-05-17T15:23:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 66 64 61 33 34 |..*...$...1fda34| 2024-05-17T15:23:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 38 30 2d 66 61 36 31 2d 34 62 34 62 2d 39 33 61 |80-fa61-4b4b-93a| 2024-05-17T15:23:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 36 2d 32 64 64 30 36 33 61 63 35 30 66 34 01 01 |6-2dd063ac50f4..| 2024-05-17T15:23:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12844609881d 2024-05-17T15:23:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 34 30 34 38 30 |..*...$...b40480| 2024-05-17T15:23:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 65 31 2d 33 61 65 66 2d 34 36 62 62 2d 61 38 36 |e1-3aef-46bb-a86| 2024-05-17T15:23:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 66 2d 30 35 61 31 65 65 30 64 32 64 32 36 01 01 |f-05a1ee0d2d26..| 2024-05-17T15:23:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12845014c61a 2024-05-17T15:23:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 33 36 61 33 39 31 |..*...$...36a391| 2024-05-17T15:23:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 33 64 2d 33 37 38 61 2d 34 31 32 61 2d 61 64 35 |3d-378a-412a-ad5| 2024-05-17T15:23:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 62 2d 39 63 32 36 39 38 36 30 31 37 33 65 01 01 |b-9c269860173e..| 2024-05-17T15:23:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_128479b7028b 2024-05-17T15:23:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 33 64 66 39 38 34 |..*...$...3df984| 2024-05-17T15:23:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 37 38 2d 61 34 38 39 2d 34 61 33 33 2d 62 30 31 |78-a489-4a33-b01| 2024-05-17T15:23:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 63 2d 31 33 32 66 61 38 32 34 31 35 33 34 01 01 |c-132fa8241534..| 2024-05-17T15:23:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12849a8edc8d 2024-05-17T15:23:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 64 30 62 38 30 38 |..*...$...d0b808| 2024-05-17T15:23:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 36 37 2d 32 61 33 62 2d 34 39 34 37 2d 61 65 32 |67-2a3b-4947-ae2| 2024-05-17T15:23:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 64 2d 33 30 63 64 36 62 34 65 61 38 63 63 01 01 |d-30cd6b4ea8cc..| 2024-05-17T15:23:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_1284a2612bb4 2024-05-17T15:23:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 35 30 66 30 38 |..*...$...c50f08| 2024-05-17T15:23:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 65 35 2d 65 66 32 65 2d 34 63 63 39 2d 61 66 64 |e5-ef2e-4cc9-afd| 2024-05-17T15:23:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 37 2d 66 62 31 38 64 64 33 66 30 33 64 63 01 01 |7-fb18dd3f03dc..| 2024-05-17T15:23:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_1284a94e4c87 2024-05-17T15:23:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 33 32 61 66 38 39 |..*...$...32af89| 2024-05-17T15:23:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 35 64 2d 38 62 35 66 2d 34 64 31 38 2d 39 39 64 |5d-8b5f-4d18-99d| 2024-05-17T15:23:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 62 2d 62 32 61 64 34 35 61 37 61 63 36 39 01 01 |b-b2ad45a7ac69..| 2024-05-17T15:23:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_1284ad92c3a1 2024-05-17T15:23:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 36 61 65 38 35 |..*...$...46ae85| 2024-05-17T15:23:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 35 39 2d 31 38 33 30 2d 34 63 33 64 2d 61 64 66 |59-1830-4c3d-adf| 2024-05-17T15:23:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 63 2d 63 38 66 39 36 33 34 35 65 66 61 64 01 01 |c-c8f96345efad..| 2024-05-17T15:23:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_1284c16da516 2024-05-17T15:23:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 32 39 36 38 64 62 |..*...$...2968db| 2024-05-17T15:23:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 65 64 2d 64 63 36 36 2d 34 65 38 32 2d 62 65 35 |ed-dc66-4e82-be5| 2024-05-17T15:23:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 66 2d 62 63 36 65 33 33 32 32 61 36 63 31 01 01 |f-bc6e3322a6c1..| 2024-05-17T15:23:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_1284e73f1092 2024-05-17T15:23:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 39 39 30 64 39 |..*...$...f990d9| 2024-05-17T15:23:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 33 36 2d 61 37 37 31 2d 34 38 33 62 2d 38 37 32 |36-a771-483b-872| 2024-05-17T15:23:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 38 2d 66 64 31 35 30 33 32 30 31 31 33 36 01 01 |8-fd1503201136..| 2024-05-17T15:23:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_1284f40a8353 2024-05-17T15:23:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 63 61 39 62 31 |..*...$...9ca9b1| 2024-05-17T15:23:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 36 64 2d 36 66 62 30 2d 34 36 64 34 2d 39 39 39 |6d-6fb0-46d4-999| 2024-05-17T15:23:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 38 2d 32 30 38 66 61 34 39 38 30 66 34 66 01 01 |8-208fa4980f4f..| 2024-05-17T15:23:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_1293779945bc 2024-05-17T15:23:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 63 63 39 34 32 |..*...$...9cc942| 2024-05-17T15:23:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 30 31 2d 65 33 32 34 2d 34 33 31 39 2d 61 36 34 |01-e324-4319-a64| 2024-05-17T15:23:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 36 2d 64 35 37 31 64 30 65 33 64 66 30 35 01 01 |6-d571d0e3df05..| 2024-05-17T15:23:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12939310b8e8 2024-05-17T15:23:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 39 39 39 35 61 |..*...$...99995a| 2024-05-17T15:23:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 66 61 2d 31 30 64 65 2d 34 33 38 35 2d 61 33 37 |fa-10de-4385-a37| 2024-05-17T15:23:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 39 2d 34 35 35 65 63 32 62 65 38 61 33 34 01 01 |9-455ec2be8a34..| 2024-05-17T15:23:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_1293dc4d5196 2024-05-17T15:23:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 66 62 30 63 65 34 |..*...$...fb0ce4| 2024-05-17T15:23:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 38 39 2d 33 66 36 37 2d 34 34 65 31 2d 39 37 31 |89-3f67-44e1-971| 2024-05-17T15:23:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 63 2d 30 33 62 31 30 30 35 30 37 34 61 37 01 01 |c-03b1005074a7..| 2024-05-17T15:23:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12a33ce1f7e6 2024-05-17T15:23:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 31 34 66 37 36 |..*...$...414f76| 2024-05-17T15:23:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 30 34 2d 34 32 64 30 2d 34 35 33 37 2d 62 65 61 |04-42d0-4537-bea| 2024-05-17T15:23:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 33 2d 62 66 63 66 62 65 36 36 65 32 31 30 01 01 |3-bfcfbe66e210..| 2024-05-17T15:23:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12a36c6d2380 2024-05-17T15:23:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 39 30 66 30 38 64 |..*...$...90f08d| 2024-05-17T15:23:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 31 34 2d 38 65 66 30 2d 34 62 66 66 2d 39 66 30 |14-8ef0-4bff-9f0| 2024-05-17T15:23:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 64 2d 64 38 30 34 65 64 35 39 61 61 64 31 01 01 |d-d804ed59aad1..| 2024-05-17T15:23:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12a39a9f7909 2024-05-17T15:23:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 34 65 39 65 30 |..*...$...84e9e0| 2024-05-17T15:23:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 35 62 2d 35 30 33 31 2d 34 32 30 31 2d 39 33 61 |5b-5031-4201-93a| 2024-05-17T15:23:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 62 2d 39 31 35 63 66 38 61 33 63 38 38 39 01 01 |b-915cf8a3c889..| 2024-05-17T15:23:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12a3a111b8cb 2024-05-17T15:23:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 61 61 65 32 31 31 |..*...$...aae211| 2024-05-17T15:23:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 34 32 2d 62 66 39 66 2d 34 64 39 37 2d 39 61 64 |42-bf9f-4d97-9ad| 2024-05-17T15:23:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 37 2d 38 65 34 36 37 39 65 65 30 33 37 61 01 01 |7-8e4679ee037a..| 2024-05-17T15:23:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12a3a48a4c32 2024-05-17T15:23:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 63 36 33 39 31 |..*...$...8c6391| 2024-05-17T15:23:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 37 32 2d 36 38 31 30 2d 34 61 32 30 2d 61 35 65 |72-6810-4a20-a5e| 2024-05-17T15:23:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 65 2d 31 39 62 35 61 35 30 65 63 36 66 64 01 01 |e-19b5a50ec6fd..| 2024-05-17T15:23:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12a3ab3a6e1a 2024-05-17T15:23:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 64 39 35 64 34 66 |..*...$...d95d4f| 2024-05-17T15:23:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 33 33 2d 64 65 63 61 2d 34 34 36 64 2d 38 38 63 |33-deca-446d-88c| 2024-05-17T15:23:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 61 2d 32 66 64 34 65 66 35 36 66 35 35 34 01 01 |a-2fd4ef56f554..| 2024-05-17T15:23:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12a3ba5aa645 2024-05-17T15:23:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 35 66 34 66 33 31 |..*...$...5f4f31| 2024-05-17T15:23:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 33 62 2d 64 39 64 66 2d 34 32 30 66 2d 38 63 39 |3b-d9df-420f-8c9| 2024-05-17T15:23:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 32 2d 32 64 39 65 34 38 31 31 38 34 62 63 01 01 |2-2d9e481184bc..| 2024-05-17T15:23:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12ad1f2c19f8 2024-05-17T15:23:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 62 38 31 36 35 62 |..*...$...b8165b| 2024-05-17T15:23:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 35 34 2d 65 63 64 61 2d 34 63 61 33 2d 38 64 30 |54-ecda-4ca3-8d0| 2024-05-17T15:23:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 61 2d 66 30 63 31 61 38 35 31 31 32 63 37 01 01 |a-f0c1a85112c7..| 2024-05-17T15:23:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12ad2ccfa334 2024-05-17T15:23:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 61 62 64 62 35 |..*...$...4abdb5| 2024-05-17T15:23:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 32 31 2d 61 66 61 65 2d 34 33 33 34 2d 61 64 31 |21-afae-4334-ad1| 2024-05-17T15:23:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 33 2d 37 30 38 31 35 34 32 30 65 62 61 38 01 01 |3-70815420eba8..| 2024-05-17T15:23:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12ad5935de3e 2024-05-17T15:23:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 34 35 32 33 66 |..*...$...84523f| 2024-05-17T15:23:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 63 31 2d 32 38 30 32 2d 34 38 36 38 2d 39 62 36 |c1-2802-4868-9b6| 2024-05-17T15:23:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 38 2d 31 33 64 31 30 65 39 62 65 37 65 32 01 01 |8-13d10e9be7e2..| 2024-05-17T15:23:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12adc2237663 2024-05-17T15:23:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 30 38 33 35 66 66 |..*...$...0835ff| 2024-05-17T15:23:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 66 35 2d 31 62 65 32 2d 34 34 39 61 2d 38 66 66 |f5-1be2-449a-8ff| 2024-05-17T15:23:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 36 2d 38 35 61 35 65 64 36 62 37 31 66 38 01 01 |6-85a5ed6b71f8..| 2024-05-17T15:23:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12adc6efeed4 2024-05-17T15:23:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 32 31 39 66 65 31 |..*...$...219fe1| 2024-05-17T15:23:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 62 64 2d 35 30 33 61 2d 34 33 37 37 2d 62 31 38 |bd-503a-4377-b18| 2024-05-17T15:23:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 37 2d 66 38 33 30 36 32 66 35 62 31 62 61 01 01 |7-f83062f5b1ba..| 2024-05-17T15:23:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_12ade767e69f 2024-05-17T15:23:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (48 bytes) : 2024-05-17T15:23:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 32 64 37 64 64 |..*...$...42d7dd| 2024-05-17T15:23:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 37 37 2d 66 35 65 31 2d 34 63 31 39 2d 61 37 39 |77-f5e1-4c19-a79| 2024-05-17T15:23:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 30 2d 65 32 30 37 33 66 35 62 36 37 65 65 01 01 |0-e2073f5b67ee..| 2024-05-17T15:23:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 2024-05-17T15:23:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:mirror_mode 2024-05-17T15:23:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (4 bytes) : 2024-05-17T15:23:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 00 00 00 |....| 2024-05-17T15:23:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000004 2024-05-17T15:23:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:mirror_peer_19bb688f-aeae-4d87-ba1c-7b8f4695256d 2024-05-17T15:23:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (132 bytes) : 2024-05-17T15:23:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 31 39 62 62 36 38 |..~...$...19bb68| 2024-05-17T15:23:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 38 66 2d 61 65 61 65 2d 34 64 38 37 2d 62 61 31 |8f-aeae-4d87-ba1| 2024-05-17T15:23:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 63 2d 37 62 38 66 34 36 39 35 32 35 36 64 08 00 |c-7b8f4695256d..| 2024-05-17T15:23:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 63 6c 75 73 74 65 72 32 0d 00 00 00 63 6c |..cluster2....cl| 2024-05-17T15:23:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2024-05-17T15:23:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 ff ff ff 02 24 00 00 00 37 65 38 31 36 31 32 32 |....$...7e816122| 2024-05-17T15:23:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 2d 63 31 31 65 2d 34 61 35 34 2d 39 62 61 62 2d |-c11e-4a54-9bab-| 2024-05-17T15:23:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 37 33 37 32 66 34 30 36 32 31 33 62 69 76 47 66 |7372f406213bivGf| 2024-05-17T15:23:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 a6 61 1f 0e |.a..| 2024-05-17T15:23:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000084 2024-05-17T15:23:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:mirror_uuid 2024-05-17T15:23:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 63 66 30 30 32 31 35 65 2d 36 38 63 34 2d 34 39 |cf00215e-68c4-49| 2024-05-17T15:23:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 33 38 2d 38 61 64 38 2d 39 62 61 32 30 30 63 38 |38-8ad8-9ba200c8| 2024-05-17T15:23:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 61 35 63 64 |a5cd| 2024-05-17T15:23:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_0218530c-9285-4e1b-ac61-baf8583aceb9_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 29 60 48 53 10 00 |..........)`HS..| 2024-05-17T15:23:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 49 df 3f 08 00 |I.?..| 2024-05-17T15:23:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_0835fff5-1be2-449a-8ff6-85a5ed6b71f8_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 29 60 48 53 10 00 |..........)`HS..| 2024-05-17T15:23:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 80 b6 40 08 00 |..@..| 2024-05-17T15:23:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_0e1e48a7-bf0e-42d2-9d09-07eeeeaaf56e_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c6 e0 2e 01 10 00 |................| 2024-05-17T15:23:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 16 5a 7b 0a 00 |.Z{..| 2024-05-17T15:23:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_1fda3480-fa61-4b4b-93a6-2dd063ac50f4_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 29 60 48 53 10 00 |..........)`HS..| 2024-05-17T15:23:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 55 52 41 08 00 |URA..| 2024-05-17T15:23:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_2031e3c2-db04-4d88-a9f5-2a8ddf1e9fce_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 29 60 48 53 10 00 |..........)`HS..| 2024-05-17T15:23:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 6e e2 41 08 00 |n.A..| 2024-05-17T15:23:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_219fe1bd-503a-4377-b187-f83062f5b1ba_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 29 60 48 53 10 00 |..........)`HS..| 2024-05-17T15:23:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 1a ac 42 08 00 |..B..| 2024-05-17T15:23:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_2968dbed-dc66-4e82-be5f-bc6e3322a6c1_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 29 60 48 53 10 00 |..........)`HS..| 2024-05-17T15:23:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 8b 30 43 08 00 |.0C..| 2024-05-17T15:23:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_32af895d-8b5f-4d18-99db-b2ad45a7ac69_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 94 14 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 52 d7 e6 82 10 00 |..........R.....| 2024-05-17T15:23:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 57 76 47 66 |rting replayWvGf| 2024-05-17T15:23:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 92 d8 95 07 00 |.....| 2024-05-17T15:23:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_36a3913d-378a-412a-ad5b-9c269860173e_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 29 60 48 53 10 00 |..........)`HS..| 2024-05-17T15:23:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 11 b2 43 08 00 |..C..| 2024-05-17T15:23:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_3df98478-a489-4a33-b01c-132fa8241534_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c6 e0 2e 01 10 00 |................| 2024-05-17T15:23:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 09 9e 7c 0a 00 |..|..| 2024-05-17T15:23:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_414f7604-42d0-4537-bea3-bfcfbe66e210_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c6 e0 2e 01 10 00 |................| 2024-05-17T15:23:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 d7 b1 7d 0a 00 |..}..| 2024-05-17T15:23:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_42d7dd77-f5e1-4c19-a790-e2073f5b67ee_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 94 14 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 52 d7 e6 82 10 00 |..........R.....| 2024-05-17T15:23:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 57 76 47 66 |rting replayWvGf| 2024-05-17T15:23:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 e3 44 97 07 00 |.D...| 2024-05-17T15:23:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_46ae8559-1830-4c3d-adfc-c8f96345efad_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 29 60 48 53 10 00 |..........)`HS..| 2024-05-17T15:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 1f 36 44 08 00 |.6D..| 2024-05-17T15:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_4abdb521-afae-4334-ad13-70815420eba8_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 94 14 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 52 d7 e6 82 10 00 |..........R.....| 2024-05-17T15:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 57 76 47 66 |rting replayWvGf| 2024-05-17T15:23:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 79 03 98 07 00 |y....| 2024-05-17T15:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_5191746b-1578-4957-a477-41bde526d1e0_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b3 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 3f f3 e9 10 00 |...........?....| 2024-05-17T15:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 50 76 47 66 |rting replayPvGf| 2024-05-17T15:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 e5 9d d1 05 00 |.....| 2024-05-17T15:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_59928e42-fdf9-4cdf-a1c9-1cabd73a05c5_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b3 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 3f f3 e9 10 00 |...........?....| 2024-05-17T15:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 50 76 47 66 |rting replayPvGf| 2024-05-17T15:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 9d 0a d3 05 00 |.....| 2024-05-17T15:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_5f4f313b-d9df-420f-8c92-2d9e481184bc_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b3 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 3f f3 e9 10 00 |...........?....| 2024-05-17T15:23:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 50 76 47 66 |rting replayPvGf| 2024-05-17T15:23:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 e9 c4 d3 05 00 |.....| 2024-05-17T15:23:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_6e4b54d1-2cee-4ccb-aef6-8e1d0ba0e670_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c6 e0 2e 01 10 00 |................| 2024-05-17T15:23:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 f2 d3 7e 0a 00 |..~..| 2024-05-17T15:23:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_75a0868a-edaf-438e-a7e1-fedf558dccfa_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c6 e0 2e 01 10 00 |................| 2024-05-17T15:23:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 71 d8 7f 0a 00 |q....| 2024-05-17T15:23:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_76d62c45-9c33-4b38-aeb9-d12dc0c2247a_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 29 60 48 53 10 00 |..........)`HS..| 2024-05-17T15:23:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 08 ba 44 08 00 |..D..| 2024-05-17T15:23:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_7b475e11-a35f-4917-9f29-282df00e1098_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b3 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 3f f3 e9 10 00 |...........?....| 2024-05-17T15:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 50 76 47 66 |rting replayPvGf| 2024-05-17T15:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 b0 67 d4 05 00 |.g...| 2024-05-17T15:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_7fede8a9-d664-48e9-b43a-740b49fc17be_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 29 60 48 53 10 00 |..........)`HS..| 2024-05-17T15:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 98 3a 45 08 00 |.:E..| 2024-05-17T15:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_84523fc1-2802-4868-9b68-13d10e9be7e2_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b3 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 3f f3 e9 10 00 |...........?....| 2024-05-17T15:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 50 76 47 66 |rting replayPvGf| 2024-05-17T15:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 3b 09 d5 05 00 |;....| 2024-05-17T15:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_84e9e05b-5031-4201-93ab-915cf8a3c889_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c6 e0 2e 01 10 00 |................| 2024-05-17T15:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 36 e6 80 0a 00 |6....| 2024-05-17T15:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_8d44a27c-5bdd-4a78-9b8a-9a4d22d34367_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b3 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 3f f3 e9 10 00 |...........?....| 2024-05-17T15:23:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 50 76 47 66 |rting replayPvGf| 2024-05-17T15:23:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 55 ce d5 05 00 |U....| 2024-05-17T15:23:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_90f08d14-8ef0-4bff-9f0d-d804ed59aad1_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b3 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 3f f3 e9 10 00 |...........?....| 2024-05-17T15:23:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 50 76 47 66 |rting replayPvGf| 2024-05-17T15:23:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 43 5d d6 05 00 |C]...| 2024-05-17T15:23:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_95950a54-837a-461f-aafc-36eac3b2a17a_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c6 e0 2e 01 10 00 |................| 2024-05-17T15:23:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 55 dd 81 0a 00 |U....| 2024-05-17T15:23:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_9ca9b16d-6fb0-46d4-9998-208fa4980f4f_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c6 e0 2e 01 10 00 |................| 2024-05-17T15:23:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 56 d6 82 0a 00 |V....| 2024-05-17T15:23:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_9cc94201-e324-4319-a646-d571d0e3df05_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 29 60 48 53 10 00 |..........)`HS..| 2024-05-17T15:23:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 93 b9 45 08 00 |..E..| 2024-05-17T15:23:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_9d50b403-8aa5-4da6-8e72-4a3f61c9358b_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 29 60 48 53 10 00 |..........)`HS..| 2024-05-17T15:23:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 14 3d 46 08 00 |.=F..| 2024-05-17T15:23:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_9e41e805-f938-4777-a0dd-ce32b1acfcf1_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c6 e0 2e 01 10 00 |................| 2024-05-17T15:23:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 3c ec 83 0a 00 |<....| 2024-05-17T15:23:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_b8165b54-ecda-4ca3-8d0a-f0c1a85112c7_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b3 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 3f f3 e9 10 00 |...........?....| 2024-05-17T15:23:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 50 76 47 66 |rting replayPvGf| 2024-05-17T15:23:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 df f1 d6 05 00 |.....| 2024-05-17T15:23:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_bef15777-4cd7-47fb-8829-12dabc805b90_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c6 e0 2e 01 10 00 |................| 2024-05-17T15:23:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 42 35 85 0a 00 |B5...| 2024-05-17T15:23:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_c50f08e5-ef2e-4cc9-afd7-fb18dd3f03dc_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c6 e0 2e 01 10 00 |................| 2024-05-17T15:23:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 2d 33 86 0a 00 |-3...| 2024-05-17T15:23:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_ceaa7752-4708-4453-aad7-8c9fe4b5ad62_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c6 e0 2e 01 10 00 |................| 2024-05-17T15:23:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 69 76 47 66 |rting replayivGf| 2024-05-17T15:23:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 ef 3d 87 0a 00 |.=...| 2024-05-17T15:23:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_e3330572-d02a-4104-a14f-d83973929a2c_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b3 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 3f f3 e9 10 00 |...........?....| 2024-05-17T15:23:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 50 76 47 66 |rting replayPvGf| 2024-05-17T15:23:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 1b 90 d7 05 00 |.....| 2024-05-17T15:23:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:remote_status_global_fb0ce489-3f67-44e1-971c-03b1005074a7_7e816122-c11e-4a54-9bab-7372f406213b 2024-05-17T15:23:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (85 bytes) : 2024-05-17T15:23:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 b3 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 e8 3f f3 e9 10 00 |...........?....| 2024-05-17T15:23:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 1d 00 00 00 03 0f 00 00 00 73 74 61 |.............sta| 2024-05-17T15:23:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 72 74 69 6e 67 20 72 65 70 6c 61 79 50 76 47 66 |rting replayPvGf| 2024-05-17T15:23:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 3d ad d8 05 00 |=....| 2024-05-17T15:23:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000055 2024-05-17T15:23:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_0218530c-9285-4e1b-ac61-baf8583aceb9 2024-05-17T15:23:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 31 14 00 00 00 00 00 00 01 |..,....1........| 2024-05-17T15:23:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 10 e6 60 49 10 00 |............`I..| 2024-05-17T15:23:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 36 33 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |58635,"remote_sn| 2024-05-17T15:23:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 36 33 35 2c 22 72 65 |":1715958635,"re| 2024-05-17T15:23:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 66 76 47 66 ff a0 c2 06 00 |e"}fvGf.....| 2024-05-17T15:23:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_0835fff5-1be2-449a-8ff6-85a5ed6b71f8 2024-05-17T15:23:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 c9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 85 f1 ef b5 10 00 |................| 2024-05-17T15:23:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 36 32 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |58626,"remote_sn| 2024-05-17T15:23:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 36 32 36 2c 22 72 65 |":1715958626,"re| 2024-05-17T15:23:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 5f 76 47 66 47 ea 7f 06 00 |e"}_vGfG....| 2024-05-17T15:23:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_0e1e48a7-bf0e-42d2-9d09-07eeeeaaf56e 2024-05-17T15:23:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 7a 13 00 00 00 00 00 00 01 |..,....z........| 2024-05-17T15:23:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ba 29 03 f4 10 00 |...........)....| 2024-05-17T15:23:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 39 34 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |58940,"remote_sn| 2024-05-17T15:23:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 39 34 30 2c 22 72 65 |":1715958940,"re| 2024-05-17T15:23:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 64 76 47 66 77 65 4b 06 00 |e"}dvGfweK..| 2024-05-17T15:23:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_1cbb263d-bbcf-4037-8f80-52c0b1dfd284 2024-05-17T15:23:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 31 14 00 00 00 00 00 00 01 |..,....1........| 2024-05-17T15:23:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 10 e6 60 49 10 00 |............`I..| 2024-05-17T15:23:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 36 38 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |58682,"remote_sn| 2024-05-17T15:23:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 36 38 32 2c 22 72 65 |":1715958682,"re| 2024-05-17T15:23:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 66 76 47 66 9e 87 c3 06 00 |e"}fvGf.....| 2024-05-17T15:23:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_1fda3480-fa61-4b4b-93a6-2dd063ac50f4 2024-05-17T15:23:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 7a 13 00 00 00 00 00 00 01 |..,....z........| 2024-05-17T15:23:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ba 29 03 f4 10 00 |...........)....| 2024-05-17T15:23:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 38 36 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |58867,"remote_sn| 2024-05-17T15:23:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 38 36 37 2c 22 72 65 |":1715958867,"re| 2024-05-17T15:23:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 64 76 47 66 f2 d3 4c 06 00 |e"}dvGf..L..| 2024-05-17T15:23:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_2031e3c2-db04-4d88-a9f5-2a8ddf1e9fce 2024-05-17T15:23:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 c9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 85 f1 ef b5 10 00 |................| 2024-05-17T15:23:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 35 39 38 2c 22 72 65 6d 6f 74 65 5f 73 6e |58598,"remote_sn| 2024-05-17T15:23:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 35 39 38 2c 22 72 65 |":1715958598,"re| 2024-05-17T15:23:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 5f 76 47 66 67 98 81 06 00 |e"}_vGfg....| 2024-05-17T15:23:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_219fe1bd-503a-4377-b187-f83062f5b1ba 2024-05-17T15:23:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 c9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 85 f1 ef b5 10 00 |................| 2024-05-17T15:23:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 36 31 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |58611,"remote_sn| 2024-05-17T15:23:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 36 31 31 2c 22 72 65 |":1715958611,"re| 2024-05-17T15:23:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 5f 76 47 66 94 47 82 06 00 |e"}_vGf.G...| 2024-05-17T15:23:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_2968dbed-dc66-4e82-be5f-bc6e3322a6c1 2024-05-17T15:23:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (300 bytes) : 2024-05-17T15:23:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 c9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 85 f1 ef b5 10 00 |................| 2024-05-17T15:23:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 31 33 34 30 37 37 34 34 30 2e 30 |hot":134077440.0| 2024-05-17T15:23:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 30 37 37 34 34 30 |bytes":134077440| 2024-05-17T15:23:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 35 39 30 36 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |59063,"remote_sn| 2024-05-17T15:23:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 22 3a 31 37 31 35 39 35 39 30 36 33 2c 22 72 65 |":1715959063,"re| 2024-05-17T15:23:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000120 65 22 7d 5f 76 47 66 6e f3 82 06 00 |e"}_vGfn....| 2024-05-17T15:23:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000012c 2024-05-17T15:23:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_32af895d-8b5f-4d18-99db-b2ad45a7ac69 2024-05-17T15:23:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 31 14 00 00 00 00 00 00 01 |..,....1........| 2024-05-17T15:23:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 10 e6 60 49 10 00 |............`I..| 2024-05-17T15:23:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 36 34 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |58642,"remote_sn| 2024-05-17T15:23:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 36 34 32 2c 22 72 65 |":1715958642,"re| 2024-05-17T15:23:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 66 76 47 66 4a 21 c4 06 00 |e"}fvGfJ!...| 2024-05-17T15:23:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_36a3913d-378a-412a-ad5b-9c269860173e 2024-05-17T15:23:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 84 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 89 bc ba 75 10 00 |.............u..| 2024-05-17T15:23:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 37 36 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |58764,"remote_sn| 2024-05-17T15:23:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 37 36 34 2c 22 72 65 |":1715958764,"re| 2024-05-17T15:23:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 6b 76 47 66 20 7d e4 02 00 |e"}kvGf }...| 2024-05-17T15:23:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_3df98478-a489-4a33-b01c-132fa8241534 2024-05-17T15:23:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 c9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 85 f1 ef b5 10 00 |................| 2024-05-17T15:23:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 39 30 30 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |59001,"remote_sn| 2024-05-17T15:23:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 39 30 30 31 2c 22 72 65 |":1715959001,"re| 2024-05-17T15:23:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 5f 76 47 66 c2 ac 83 06 00 |e"}_vGf.....| 2024-05-17T15:23:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_414f7604-42d0-4537-bea3-bfcfbe66e210 2024-05-17T15:23:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 7a 13 00 00 00 00 00 00 01 |..,....z........| 2024-05-17T15:23:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ba 29 03 f4 10 00 |...........)....| 2024-05-17T15:23:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 37 33 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |58736,"remote_sn| 2024-05-17T15:23:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 37 33 36 2c 22 72 65 |":1715958736,"re| 2024-05-17T15:23:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 64 76 47 66 dc be 4d 06 00 |e"}dvGf..M..| 2024-05-17T15:23:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_42d7dd77-f5e1-4c19-a790-e2073f5b67ee 2024-05-17T15:23:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 c9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 85 f1 ef b5 10 00 |................| 2024-05-17T15:23:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 37 34 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |58742,"remote_sn| 2024-05-17T15:23:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 37 34 32 2c 22 72 65 |":1715958742,"re| 2024-05-17T15:23:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 5f 76 47 66 fe 95 84 06 00 |e"}_vGf.....| 2024-05-17T15:23:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_46ae8559-1830-4c3d-adfc-c8f96345efad 2024-05-17T15:23:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 84 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 89 bc ba 75 10 00 |.............u..| 2024-05-17T15:23:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 37 32 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |58729,"remote_sn| 2024-05-17T15:23:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 37 32 39 2c 22 72 65 |":1715958729,"re| 2024-05-17T15:23:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 6b 76 47 66 56 60 e5 02 00 |e"}kvGfV`...| 2024-05-17T15:23:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_4abdb521-afae-4334-ad13-70815420eba8 2024-05-17T15:23:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 31 14 00 00 00 00 00 00 01 |..,....1........| 2024-05-17T15:23:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 10 e6 60 49 10 00 |............`I..| 2024-05-17T15:23:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 37 30 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |58709,"remote_sn| 2024-05-17T15:23:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 37 30 39 2c 22 72 65 |":1715958709,"re| 2024-05-17T15:23:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 66 76 47 66 70 f4 c4 06 00 |e"}fvGfp....| 2024-05-17T15:23:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_5191746b-1578-4957-a477-41bde526d1e0 2024-05-17T15:23:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 84 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 89 bc ba 75 10 00 |.............u..| 2024-05-17T15:23:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 38 35 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |58859,"remote_sn| 2024-05-17T15:23:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 38 35 39 2c 22 72 65 |":1715958859,"re| 2024-05-17T15:23:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 6b 76 47 66 0f 35 e6 02 00 |e"}kvGf.5...| 2024-05-17T15:23:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_59928e42-fdf9-4cdf-a1c9-1cabd73a05c5 2024-05-17T15:23:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 7a 13 00 00 00 00 00 00 01 |..,....z........| 2024-05-17T15:23:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ba 29 03 f4 10 00 |...........)....| 2024-05-17T15:23:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 35 38 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |58585,"remote_sn| 2024-05-17T15:23:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 35 38 35 2c 22 72 65 |":1715958585,"re| 2024-05-17T15:23:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 64 76 47 66 c7 6b 4e 06 00 |e"}dvGf.kN..| 2024-05-17T15:23:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_5f4f313b-d9df-420f-8c92-2d9e481184bc 2024-05-17T15:23:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (300 bytes) : 2024-05-17T15:23:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 7a 13 00 00 00 00 00 00 01 |..,....z........| 2024-05-17T15:23:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ba 29 03 f4 10 00 |...........)....| 2024-05-17T15:23:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 31 33 33 38 38 31 38 35 36 2e 30 |hot":133881856.0| 2024-05-17T15:23:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 38 38 31 38 35 36 |bytes":133881856| 2024-05-17T15:23:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 35 39 30 37 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |59079,"remote_sn| 2024-05-17T15:23:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 22 3a 31 37 31 35 39 35 39 30 37 39 2c 22 72 65 |":1715959079,"re| 2024-05-17T15:23:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000120 65 22 7d 64 76 47 66 63 1c 4f 06 00 |e"}dvGfc.O..| 2024-05-17T15:23:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000012c 2024-05-17T15:23:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_69d8440b-93d1-40e8-b675-4ab5105af9bd 2024-05-17T15:23:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 31 14 00 00 00 00 00 00 01 |..,....1........| 2024-05-17T15:23:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 10 e6 60 49 10 00 |............`I..| 2024-05-17T15:23:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 37 32 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |58721,"remote_sn| 2024-05-17T15:23:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 37 32 31 2c 22 72 65 |":1715958721,"re| 2024-05-17T15:23:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 66 76 47 66 4a 81 c5 06 00 |e"}fvGfJ....| 2024-05-17T15:23:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_6c3aab86-8ad4-4a10-8f48-3d66ec7f5b30 2024-05-17T15:23:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 c9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 85 f1 ef b5 10 00 |................| 2024-05-17T15:23:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 39 30 38 2c 22 72 65 6d 6f 74 65 5f 73 6e |58908,"remote_sn| 2024-05-17T15:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 39 30 38 2c 22 72 65 |":1715958908,"re| 2024-05-17T15:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 5f 76 47 66 60 38 85 06 00 |e"}_vGf`8...| 2024-05-17T15:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_6c86503a-19c3-470e-81c4-44311421ecfd 2024-05-17T15:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 31 14 00 00 00 00 00 00 01 |..,....1........| 2024-05-17T15:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 10 e6 60 49 10 00 |............`I..| 2024-05-17T15:23:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 37 39 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |58792,"remote_sn| 2024-05-17T15:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 37 39 32 2c 22 72 65 |":1715958792,"re| 2024-05-17T15:23:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 66 76 47 66 3e 0e c6 06 00 |e"}fvGf>....| 2024-05-17T15:23:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_6e4b54d1-2cee-4ccb-aef6-8e1d0ba0e670 2024-05-17T15:23:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 c9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 85 f1 ef b5 10 00 |................| 2024-05-17T15:23:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 38 39 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |58892,"remote_sn| 2024-05-17T15:23:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 38 39 32 2c 22 72 65 |":1715958892,"re| 2024-05-17T15:23:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 5f 76 47 66 19 d5 85 06 00 |e"}_vGf.....| 2024-05-17T15:23:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_729f0874-9d81-4812-83cf-ae9935d33036 2024-05-17T15:23:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 31 14 00 00 00 00 00 00 01 |..,....1........| 2024-05-17T15:23:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 10 e6 60 49 10 00 |............`I..| 2024-05-17T15:23:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 37 34 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |58749,"remote_sn| 2024-05-17T15:23:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 37 34 39 2c 22 72 65 |":1715958749,"re| 2024-05-17T15:23:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 66 76 47 66 ee 90 c6 06 00 |e"}fvGf.....| 2024-05-17T15:23:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_75a0868a-edaf-438e-a7e1-fedf558dccfa 2024-05-17T15:23:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 c9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 85 f1 ef b5 10 00 |................| 2024-05-17T15:23:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 39 30 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |58900,"remote_sn| 2024-05-17T15:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 39 30 30 2c 22 72 65 |":1715958900,"re| 2024-05-17T15:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 5f 76 47 66 e4 7a 86 06 00 |e"}_vGf.z...| 2024-05-17T15:23:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_76d62c45-9c33-4b38-aeb9-d12dc0c2247a 2024-05-17T15:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 c9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 85 f1 ef b5 10 00 |................| 2024-05-17T15:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 39 35 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |58955,"remote_sn| 2024-05-17T15:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 39 35 35 2c 22 72 65 |":1715958955,"re| 2024-05-17T15:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 5f 76 47 66 88 26 87 06 00 |e"}_vGf.&...| 2024-05-17T15:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_7b475e11-a35f-4917-9f29-282df00e1098 2024-05-17T15:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (300 bytes) : 2024-05-17T15:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 31 14 00 00 00 00 00 00 01 |..,....1........| 2024-05-17T15:23:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 10 e6 60 49 10 00 |............`I..| 2024-05-17T15:23:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 31 33 34 30 38 35 36 33 32 2e 30 |hot":134085632.0| 2024-05-17T15:23:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 30 38 35 36 33 32 |bytes":134085632| 2024-05-17T15:23:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 35 39 30 35 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |59055,"remote_sn| 2024-05-17T15:23:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 22 3a 31 37 31 35 39 35 39 30 35 35 2c 22 72 65 |":1715959055,"re| 2024-05-17T15:23:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000120 65 22 7d 66 76 47 66 83 1a c7 06 00 |e"}fvGf.....| 2024-05-17T15:23:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000012c 2024-05-17T15:23:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_7fede8a9-d664-48e9-b43a-740b49fc17be 2024-05-17T15:23:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (300 bytes) : 2024-05-17T15:23:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 7a 13 00 00 00 00 00 00 01 |..,....z........| 2024-05-17T15:23:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ba 29 03 f4 10 00 |...........)....| 2024-05-17T15:23:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.026 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 31 33 34 31 39 39 32 39 36 2e 30 |hot":134199296.0| 2024-05-17T15:23:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 31 39 39 32 39 36 |bytes":134199296| 2024-05-17T15:23:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 35 39 30 34 38 2c 22 72 65 6d 6f 74 65 5f 73 6e |59048,"remote_sn| 2024-05-17T15:23:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 22 3a 31 37 31 35 39 35 39 30 34 38 2c 22 72 65 |":1715959048,"re| 2024-05-17T15:23:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000120 65 22 7d 64 76 47 66 b4 bb 4f 06 00 |e"}dvGf..O..| 2024-05-17T15:23:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000012c 2024-05-17T15:23:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_82562a71-d364-4946-b5e9-67440f1071c4 2024-05-17T15:23:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 c9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 85 f1 ef b5 10 00 |................| 2024-05-17T15:23:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.028 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 38 37 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |58876,"remote_sn| 2024-05-17T15:23:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 38 37 36 2c 22 72 65 |":1715958876,"re| 2024-05-17T15:23:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 5f 76 47 66 a9 c3 87 06 00 |e"}_vGf.....| 2024-05-17T15:23:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_84523fc1-2802-4868-9b68-13d10e9be7e2 2024-05-17T15:23:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.029 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 84 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 89 bc ba 75 10 00 |.............u..| 2024-05-17T15:23:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 37 38 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |58780,"remote_sn| 2024-05-17T15:23:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 37 38 30 2c 22 72 65 |":1715958780,"re| 2024-05-17T15:23:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 6b 76 47 66 c5 b8 e6 02 00 |e"}kvGf.....| 2024-05-17T15:23:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_84e9e05b-5031-4201-93ab-915cf8a3c889 2024-05-17T15:23:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 84 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 89 bc ba 75 10 00 |.............u..| 2024-05-17T15:23:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 37 37 30 2c 22 72 65 6d 6f 74 65 5f 73 6e |58770,"remote_sn| 2024-05-17T15:23:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 37 37 30 2c 22 72 65 |":1715958770,"re| 2024-05-17T15:23:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 6b 76 47 66 f5 3f e7 02 00 |e"}kvGf.?...| 2024-05-17T15:23:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_8c639172-6810-4a20-a5ee-19b5a50ec6fd 2024-05-17T15:23:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 84 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 89 bc ba 75 10 00 |.............u..| 2024-05-17T15:23:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 36 35 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |58653,"remote_sn| 2024-05-17T15:23:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 36 35 33 2c 22 72 65 |":1715958653,"re| 2024-05-17T15:23:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 6b 76 47 66 2c c3 e7 02 00 |e"}kvGf,....| 2024-05-17T15:23:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_8d44a27c-5bdd-4a78-9b8a-9a4d22d34367 2024-05-17T15:23:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 c9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 85 f1 ef b5 10 00 |................| 2024-05-17T15:23:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 38 35 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |58851,"remote_sn| 2024-05-17T15:23:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 38 35 31 2c 22 72 65 |":1715958851,"re| 2024-05-17T15:23:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 5f 76 47 66 75 62 88 06 00 |e"}_vGfub...| 2024-05-17T15:23:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_90f08d14-8ef0-4bff-9f0d-d804ed59aad1 2024-05-17T15:23:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 7a 13 00 00 00 00 00 00 01 |..,....z........| 2024-05-17T15:23:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ba 29 03 f4 10 00 |...........)....| 2024-05-17T15:23:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 35 37 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |58579,"remote_sn| 2024-05-17T15:23:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 35 37 39 2c 22 72 65 |":1715958579,"re| 2024-05-17T15:23:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 64 76 47 66 c5 5c 50 06 00 |e"}dvGf.\P..| 2024-05-17T15:23:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_95950a54-837a-461f-aafc-36eac3b2a17a 2024-05-17T15:23:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 31 14 00 00 00 00 00 00 01 |..,....1........| 2024-05-17T15:23:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 10 e6 60 49 10 00 |............`I..| 2024-05-17T15:23:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 39 31 36 2c 22 72 65 6d 6f 74 65 5f 73 6e |58916,"remote_sn| 2024-05-17T15:23:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 39 31 36 2c 22 72 65 |":1715958916,"re| 2024-05-17T15:23:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 66 76 47 66 61 9a c7 06 00 |e"}fvGfa....| 2024-05-17T15:23:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_99995afa-10de-4385-a379-455ec2be8a34 2024-05-17T15:23:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (300 bytes) : 2024-05-17T15:23:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 84 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 89 bc ba 75 10 00 |.............u..| 2024-05-17T15:23:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 31 33 31 39 37 33 31 32 30 2e 30 |hot":131973120.0| 2024-05-17T15:23:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 62 79 74 65 73 22 3a 31 33 31 39 37 33 31 32 30 |bytes":131973120| 2024-05-17T15:23:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 35 39 30 34 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |59041,"remote_sn| 2024-05-17T15:23:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 22 3a 31 37 31 35 39 35 39 30 34 31 2c 22 72 65 |":1715959041,"re| 2024-05-17T15:23:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000120 65 22 7d 6b 76 47 66 fb 46 e8 02 00 |e"}kvGf.F...| 2024-05-17T15:23:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000012c 2024-05-17T15:23:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_9ca9b16d-6fb0-46d4-9998-208fa4980f4f 2024-05-17T15:23:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 84 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 89 bc ba 75 10 00 |.............u..| 2024-05-17T15:23:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 38 30 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |58802,"remote_sn| 2024-05-17T15:23:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 38 30 32 2c 22 72 65 |":1715958802,"re| 2024-05-17T15:23:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 6b 76 47 66 d7 c6 e8 02 00 |e"}kvGf.....| 2024-05-17T15:23:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_9cc94201-e324-4319-a646-d571d0e3df05 2024-05-17T15:23:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 84 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 89 bc ba 75 10 00 |.............u..| 2024-05-17T15:23:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 39 37 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |58977,"remote_sn| 2024-05-17T15:23:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 39 37 37 2c 22 72 65 |":1715958977,"re| 2024-05-17T15:23:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 6b 76 47 66 fb 45 e9 02 00 |e"}kvGf.E...| 2024-05-17T15:23:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_9d50b403-8aa5-4da6-8e72-4a3f61c9358b 2024-05-17T15:23:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (300 bytes) : 2024-05-17T15:23:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 31 14 00 00 00 00 00 00 01 |..,....1........| 2024-05-17T15:23:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 10 e6 60 49 10 00 |............`I..| 2024-05-17T15:23:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 31 33 34 31 33 35 38 30 38 2e 30 |hot":134135808.0| 2024-05-17T15:23:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 62 79 74 65 73 22 3a 31 33 34 31 33 35 38 30 38 |bytes":134135808| 2024-05-17T15:23:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 35 39 30 38 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |59087,"remote_sn| 2024-05-17T15:23:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 22 3a 31 37 31 35 39 35 39 30 38 37 2c 22 72 65 |":1715959087,"re| 2024-05-17T15:23:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000120 65 22 7d 66 76 47 66 f8 6c c8 06 00 |e"}fvGf.l...| 2024-05-17T15:23:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000012c 2024-05-17T15:23:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_9e41e805-f938-4777-a0dd-ce32b1acfcf1 2024-05-17T15:23:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 7a 13 00 00 00 00 00 00 01 |..,....z........| 2024-05-17T15:23:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ba 29 03 f4 10 00 |...........)....| 2024-05-17T15:23:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 39 36 32 2c 22 72 65 6d 6f 74 65 5f 73 6e |58962,"remote_sn| 2024-05-17T15:23:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 39 36 32 2c 22 72 65 |":1715958962,"re| 2024-05-17T15:23:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 64 76 47 66 35 f0 50 06 00 |e"}dvGf5.P..| 2024-05-17T15:23:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_aae21142-bf9f-4d97-9ad7-8e4679ee037a 2024-05-17T15:23:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 31 14 00 00 00 00 00 00 01 |..,....1........| 2024-05-17T15:23:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 10 e6 60 49 10 00 |............`I..| 2024-05-17T15:23:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 37 30 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |58703,"remote_sn| 2024-05-17T15:23:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 37 30 33 2c 22 72 65 |":1715958703,"re| 2024-05-17T15:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 66 76 47 66 2d f7 c9 06 00 |e"}fvGf-....| 2024-05-17T15:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_b40480e1-3aef-46bb-a86f-05a1ee0d2d26 2024-05-17T15:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 7a 13 00 00 00 00 00 00 01 |..,....z........| 2024-05-17T15:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ba 29 03 f4 10 00 |...........)....| 2024-05-17T15:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 38 38 34 2c 22 72 65 6d 6f 74 65 5f 73 6e |58884,"remote_sn| 2024-05-17T15:23:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 38 38 34 2c 22 72 65 |":1715958884,"re| 2024-05-17T15:23:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 64 76 47 66 37 9f 51 06 00 |e"}dvGf7.Q..| 2024-05-17T15:23:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_b8165b54-ecda-4ca3-8d0a-f0c1a85112c7 2024-05-17T15:23:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 84 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 89 bc ba 75 10 00 |.............u..| 2024-05-17T15:23:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 36 36 37 2c 22 72 65 6d 6f 74 65 5f 73 6e |58667,"remote_sn| 2024-05-17T15:23:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 36 36 37 2c 22 72 65 |":1715958667,"re| 2024-05-17T15:23:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 6b 76 47 66 26 c6 e9 02 00 |e"}kvGf&....| 2024-05-17T15:23:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_bef15777-4cd7-47fb-8829-12dabc805b90 2024-05-17T15:23:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 7a 13 00 00 00 00 00 00 01 |..,....z........| 2024-05-17T15:23:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ba 29 03 f4 10 00 |...........)....| 2024-05-17T15:23:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 39 38 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |58985,"remote_sn| 2024-05-17T15:23:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 39 38 35 2c 22 72 65 |":1715958985,"re| 2024-05-17T15:23:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 64 76 47 66 bb a9 52 06 00 |e"}dvGf..R..| 2024-05-17T15:23:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_c50f08e5-ef2e-4cc9-afd7-fb18dd3f03dc 2024-05-17T15:23:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 7a 13 00 00 00 00 00 00 01 |..,....z........| 2024-05-17T15:23:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ba 29 03 f4 10 00 |...........)....| 2024-05-17T15:23:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 39 34 38 2c 22 72 65 6d 6f 74 65 5f 73 6e |58948,"remote_sn| 2024-05-17T15:23:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 39 34 38 2c 22 72 65 |":1715958948,"re| 2024-05-17T15:23:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 64 76 47 66 39 51 53 06 00 |e"}dvGf9QS..| 2024-05-17T15:23:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_ceaa7752-4708-4453-aad7-8c9fe4b5ad62 2024-05-17T15:23:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 7a 13 00 00 00 00 00 00 01 |..,....z........| 2024-05-17T15:23:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 ba 29 03 f4 10 00 |...........)....| 2024-05-17T15:23:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 38 34 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |58843,"remote_sn| 2024-05-17T15:23:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 38 34 33 2c 22 72 65 |":1715958843,"re| 2024-05-17T15:23:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 64 76 47 66 5e ed 53 06 00 |e"}dvGf^.S..| 2024-05-17T15:23:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_d0b80867-2a3b-4947-ae2d-30cd6b4ea8cc 2024-05-17T15:23:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 31 14 00 00 00 00 00 00 01 |..,....1........| 2024-05-17T15:23:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 10 e6 60 49 10 00 |............`I..| 2024-05-17T15:23:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 35 36 35 2c 22 72 65 6d 6f 74 65 5f 73 6e |58565,"remote_sn| 2024-05-17T15:23:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 35 36 35 2c 22 72 65 |":1715958565,"re| 2024-05-17T15:23:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 66 76 47 66 66 08 cb 06 00 |e"}fvGff....| 2024-05-17T15:23:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_d95d4f33-deca-446d-88ca-2fd4ef56f554 2024-05-17T15:23:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (300 bytes) : 2024-05-17T15:23:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 84 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 89 bc ba 75 10 00 |.............u..| 2024-05-17T15:23:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 f4 00 00 00 04 e6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 31 33 33 37 30 32 36 35 36 2e 30 |hot":133702656.0| 2024-05-17T15:23:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 62 79 74 65 73 22 3a 31 33 33 37 30 32 36 35 36 |bytes":133702656| 2024-05-17T15:23:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 35 39 30 37 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |59071,"remote_sn| 2024-05-17T15:23:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 22 3a 31 37 31 35 39 35 39 30 37 31 2c 22 72 65 |":1715959071,"re| 2024-05-17T15:23:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000120 65 22 7d 6b 76 47 66 41 44 ea 02 00 |e"}kvGfAD...| 2024-05-17T15:23:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000012c 2024-05-17T15:23:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_e3330572-d02a-4104-a14f-d83973929a2c 2024-05-17T15:23:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 84 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 89 bc ba 75 10 00 |.............u..| 2024-05-17T15:23:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 39 39 33 2c 22 72 65 6d 6f 74 65 5f 73 6e |58993,"remote_sn| 2024-05-17T15:23:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 39 39 33 2c 22 72 65 |":1715958993,"re| 2024-05-17T15:23:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 6b 76 47 66 54 c6 ea 02 00 |e"}kvGfT....| 2024-05-17T15:23:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_f990d936-a771-483b-8728-fd1503201136 2024-05-17T15:23:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 84 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 89 bc ba 75 10 00 |.............u..| 2024-05-17T15:23:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 36 39 31 2c 22 72 65 6d 6f 74 65 5f 73 6e |58691,"remote_sn| 2024-05-17T15:23:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 36 39 31 2c 22 72 65 |":1715958691,"re| 2024-05-17T15:23:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 6b 76 47 66 48 44 eb 02 00 |e"}kvGfHD...| 2024-05-17T15:23:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:status_global_fb0ce489-3f67-44e1-971c-03b1005074a7 2024-05-17T15:23:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (284 bytes) : 2024-05-17T15:23:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 2c 00 00 00 08 c9 12 00 00 00 00 00 00 01 |..,.............| 2024-05-17T15:23:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 85 f1 ef b5 10 00 |................| 2024-05-17T15:23:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 02 00 00 00 ac 15 0f c3 00 00 00 00 00 00 |................| 2024-05-17T15:23:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 01 01 e4 00 00 00 04 d6 00 00 00 72 65 70 |.............rep| 2024-05-17T15:23:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2024-05-17T15:23:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2024-05-17T15:23:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 22 62 79 74 65 73 5f 70 65 72 5f 73 6e 61 70 73 |"bytes_per_snaps| 2024-05-17T15:23:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 68 6f 74 22 3a 30 2e 30 2c 22 6c 61 73 74 5f 73 |hot":0.0,"last_s| 2024-05-17T15:23:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 6e 61 70 73 68 6f 74 5f 62 79 74 65 73 22 3a 30 |napshot_bytes":0| 2024-05-17T15:23:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000090 2c 22 6c 61 73 74 5f 73 6e 61 70 73 68 6f 74 5f |,"last_snapshot_| 2024-05-17T15:23:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000a0 73 79 6e 63 5f 73 65 63 6f 6e 64 73 22 3a 30 2c |sync_seconds":0,| 2024-05-17T15:23:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000b0 22 6c 6f 63 61 6c 5f 73 6e 61 70 73 68 6f 74 5f |"local_snapshot_| 2024-05-17T15:23:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000c0 74 69 6d 65 73 74 61 6d 70 22 3a 31 37 31 35 39 |timestamp":17159| 2024-05-17T15:23:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000d0 35 38 39 36 39 2c 22 72 65 6d 6f 74 65 5f 73 6e |58969,"remote_sn| 2024-05-17T15:23:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000e0 61 70 73 68 6f 74 5f 74 69 6d 65 73 74 61 6d 70 |apshot_timestamp| 2024-05-17T15:23:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:000000f0 22 3a 31 37 31 35 39 35 38 39 36 39 2c 22 72 65 |":1715958969,"re| 2024-05-17T15:23:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000100 70 6c 61 79 5f 73 74 61 74 65 22 3a 22 69 64 6c |play_state":"idl| 2024-05-17T15:23:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000110 65 22 7d 5f 76 47 66 26 02 89 06 00 |e"}_vGf&....| 2024-05-17T15:23:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:0000011c 2024-05-17T15:23:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror rbd_mirror_leader omap vals 2024-05-17T15:23:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror rbd_mirror_leader omap vals' 2024-05-17T15:23:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirror_leader 2024-05-17T15:23:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_0218530c-9285-4e1b-ac61-baf8583aceb9 2024-05-17T15:23:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 36 39 fb 74 |..........5169.t| 2024-05-17T15:23:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 eb 90 6b 05 0a 00 00 00 01 01 04 00 00 00 |Gf..k...........| 2024-05-17T15:23:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_0835fff5-1be2-449a-8ff6-85a5ed6b71f8 2024-05-17T15:23:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 30 39 3e 74 |..........4809>t| 2024-05-17T15:23:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 d0 b9 8b 0e 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_0e1e48a7-bf0e-42d2-9d09-07eeeeaaf56e 2024-05-17T15:23:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 39 38 36 9a 74 |..........4986.t| 2024-05-17T15:23:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 e5 84 e4 12 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_1cbb263d-bbcf-4037-8f80-52c0b1dfd284 2024-05-17T15:23:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 36 39 fb 74 |..........5169.t| 2024-05-17T15:23:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 b6 bf 6b 05 0a 00 00 00 01 01 04 00 00 00 |Gf..k...........| 2024-05-17T15:23:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_1fda3480-fa61-4b4b-93a6-2dd063ac50f4 2024-05-17T15:23:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 39 38 36 9a 74 |..........4986.t| 2024-05-17T15:23:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 a5 b5 e4 12 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_2031e3c2-db04-4d88-a9f5-2a8ddf1e9fce 2024-05-17T15:23:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 30 39 3e 74 |..........4809>t| 2024-05-17T15:23:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 1a f8 8b 0e 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_219fe1bd-503a-4377-b187-f83062f5b1ba 2024-05-17T15:23:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 30 39 3e 74 |..........4809>t| 2024-05-17T15:23:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 73 14 8c 0e 0a 00 00 00 01 01 04 00 00 00 |Gfs.............| 2024-05-17T15:23:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_2968dbed-dc66-4e82-be5f-bc6e3322a6c1 2024-05-17T15:23:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 30 39 2a 74 |..........4809*t| 2024-05-17T15:23:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 f7 13 ea 0f 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_32af895d-8b5f-4d18-99db-b2ad45a7ac69 2024-05-17T15:23:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 36 39 e7 74 |..........5169.t| 2024-05-17T15:23:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 d9 e5 37 04 0a 00 00 00 01 01 04 00 00 00 |Gf..7...........| 2024-05-17T15:23:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_36a3913d-378a-412a-ad5b-9c269860173e 2024-05-17T15:23:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 30 ed 73 |..........4740.s| 2024-05-17T15:23:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 f2 9a 71 34 0a 00 00 00 01 01 04 00 00 00 |Gf..q4..........| 2024-05-17T15:23:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_3df98478-a489-4a33-b01c-132fa8241534 2024-05-17T15:23:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 30 39 2a 74 |..........4809*t| 2024-05-17T15:23:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 f5 47 ea 0f 0a 00 00 00 01 01 04 00 00 00 |Gf.G............| 2024-05-17T15:23:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_414f7604-42d0-4537-bea3-bfcfbe66e210 2024-05-17T15:23:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 39 38 36 97 74 |..........4986.t| 2024-05-17T15:23:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 10 f1 ab 11 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_42d7dd77-f5e1-4c19-a790-e2073f5b67ee 2024-05-17T15:23:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 30 39 3e 74 |..........4809>t| 2024-05-17T15:23:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 85 34 8c 0e 0a 00 00 00 01 01 04 00 00 00 |Gf.4............| 2024-05-17T15:23:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_46ae8559-1830-4c3d-adfc-c8f96345efad 2024-05-17T15:23:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 30 ca 73 |..........4740.s| 2024-05-17T15:23:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 a9 94 63 22 0a 00 00 00 01 01 04 00 00 00 |Gf..c"..........| 2024-05-17T15:23:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_4abdb521-afae-4334-ad13-70815420eba8 2024-05-17T15:23:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 36 39 fb 74 |..........5169.t| 2024-05-17T15:23:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 a5 e1 6b 05 0a 00 00 00 01 01 04 00 00 00 |Gf..k...........| 2024-05-17T15:23:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_5191746b-1578-4957-a477-41bde526d1e0 2024-05-17T15:23:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 30 3e 74 |..........4740>t| 2024-05-17T15:23:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 95 66 8c 0e 0a 00 00 00 01 01 04 00 00 00 |Gf.f............| 2024-05-17T15:23:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_59928e42-fdf9-4cdf-a1c9-1cabd73a05c5 2024-05-17T15:23:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 39 38 36 89 74 |..........4986.t| 2024-05-17T15:23:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 fa ca eb 10 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_5f4f313b-d9df-420f-8c92-2d9e481184bc 2024-05-17T15:23:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 39 38 36 97 74 |..........4986.t| 2024-05-17T15:23:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 1a 10 ac 11 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_69d8440b-93d1-40e8-b675-4ab5105af9bd 2024-05-17T15:23:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 36 39 fb 74 |..........5169.t| 2024-05-17T15:23:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 ae 07 6c 05 0a 00 00 00 01 01 04 00 00 00 |Gf..l...........| 2024-05-17T15:23:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_6c3aab86-8ad4-4a10-8f48-3d66ec7f5b30 2024-05-17T15:23:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 30 39 2a 74 |..........4809*t| 2024-05-17T15:23:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 86 69 ea 0f 0a 00 00 00 01 01 04 00 00 00 |Gf.i............| 2024-05-17T15:23:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_6c86503a-19c3-470e-81c4-44311421ecfd 2024-05-17T15:23:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 36 39 fb 74 |..........5169.t| 2024-05-17T15:23:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 1e 2e 6c 05 0a 00 00 00 01 01 04 00 00 00 |Gf..l...........| 2024-05-17T15:23:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_6e4b54d1-2cee-4ccb-aef6-8e1d0ba0e670 2024-05-17T15:23:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 30 39 2a 74 |..........4809*t| 2024-05-17T15:23:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 ab 92 ea 0f 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.080 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_729f0874-9d81-4812-83cf-ae9935d33036 2024-05-17T15:23:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 36 39 fb 74 |..........5169.t| 2024-05-17T15:23:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 17 47 6c 05 0a 00 00 00 01 01 04 00 00 00 |Gf.Gl...........| 2024-05-17T15:23:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_75a0868a-edaf-438e-a7e1-fedf558dccfa 2024-05-17T15:23:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 30 39 3e 74 |..........4809>t| 2024-05-17T15:23:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 55 83 8c 0e 0a 00 00 00 01 01 04 00 00 00 |GfU.............| 2024-05-17T15:23:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_76d62c45-9c33-4b38-aeb9-d12dc0c2247a 2024-05-17T15:23:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 30 39 2a 74 |..........4809*t| 2024-05-17T15:23:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 a3 af ea 0f 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_7b475e11-a35f-4917-9f29-282df00e1098 2024-05-17T15:23:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 36 39 fb 74 |..........5169.t| 2024-05-17T15:23:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 97 63 6c 05 0a 00 00 00 01 01 04 00 00 00 |Gf.cl...........| 2024-05-17T15:23:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_7fede8a9-d664-48e9-b43a-740b49fc17be 2024-05-17T15:23:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 39 38 36 89 74 |..........4986.t| 2024-05-17T15:23:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 20 df eb 10 0a 00 00 00 01 01 04 00 00 00 |Gf .............| 2024-05-17T15:23:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_82562a71-d364-4946-b5e9-67440f1071c4 2024-05-17T15:23:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 30 39 2a 74 |..........4809*t| 2024-05-17T15:23:25.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 38 c8 ea 0f 0a 00 00 00 01 01 04 00 00 00 |Gf8.............| 2024-05-17T15:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_84523fc1-2802-4868-9b68-13d10e9be7e2 2024-05-17T15:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 30 3e 74 |..........4740>t| 2024-05-17T15:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 d6 be 8c 0e 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_84e9e05b-5031-4201-93ab-915cf8a3c889 2024-05-17T15:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 30 fb 74 |..........4740.t| 2024-05-17T15:23:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 40 80 6c 05 0a 00 00 00 01 01 04 00 00 00 |Gf@.l...........| 2024-05-17T15:23:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_8c639172-6810-4a20-a5ee-19b5a50ec6fd 2024-05-17T15:23:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 30 97 74 |..........4740.t| 2024-05-17T15:23:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 2e 5a ac 11 0a 00 00 00 01 01 04 00 00 00 |Gf.Z............| 2024-05-17T15:23:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_8d44a27c-5bdd-4a78-9b8a-9a4d22d34367 2024-05-17T15:23:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 30 39 3e 74 |..........4809>t| 2024-05-17T15:23:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 14 d6 8c 0e 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_90f08d14-8ef0-4bff-9f0d-d804ed59aad1 2024-05-17T15:23:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 39 38 36 97 74 |..........4986.t| 2024-05-17T15:23:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 88 78 ac 11 0a 00 00 00 01 01 04 00 00 00 |Gf.x............| 2024-05-17T15:23:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_95950a54-837a-461f-aafc-36eac3b2a17a 2024-05-17T15:23:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 36 39 fb 74 |..........5169.t| 2024-05-17T15:23:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 03 9d 6c 05 0a 00 00 00 01 01 04 00 00 00 |Gf..l...........| 2024-05-17T15:23:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_99995afa-10de-4385-a379-455ec2be8a34 2024-05-17T15:23:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 30 fb 74 |..........4740.t| 2024-05-17T15:23:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 1a b6 6c 05 0a 00 00 00 01 01 04 00 00 00 |Gf..l...........| 2024-05-17T15:23:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_9ca9b16d-6fb0-46d4-9998-208fa4980f4f 2024-05-17T15:23:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 30 13 74 |..........4740.t| 2024-05-17T15:23:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 70 07 87 27 0a 00 00 00 01 01 04 00 00 00 |Gfp..'..........| 2024-05-17T15:23:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_9cc94201-e324-4319-a646-d571d0e3df05 2024-05-17T15:23:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 30 97 74 |..........4740.t| 2024-05-17T15:23:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 9e af ac 11 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_9d50b403-8aa5-4da6-8e72-4a3f61c9358b 2024-05-17T15:23:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 36 39 e7 74 |..........5169.t| 2024-05-17T15:23:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 b9 61 38 04 0a 00 00 00 01 01 04 00 00 00 |Gf.a8...........| 2024-05-17T15:23:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_9e41e805-f938-4777-a0dd-ce32b1acfcf1 2024-05-17T15:23:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 39 38 36 97 74 |..........4986.t| 2024-05-17T15:23:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 a5 c9 ac 11 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_aae21142-bf9f-4d97-9ad7-8e4679ee037a 2024-05-17T15:23:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 36 39 fb 74 |..........5169.t| 2024-05-17T15:23:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 3b d3 6c 05 0a 00 00 00 01 01 04 00 00 00 |Gf;.l...........| 2024-05-17T15:23:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_b40480e1-3aef-46bb-a86f-05a1ee0d2d26 2024-05-17T15:23:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 39 38 36 89 74 |..........4986.t| 2024-05-17T15:23:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 ad f0 eb 10 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_b8165b54-ecda-4ca3-8d0a-f0c1a85112c7 2024-05-17T15:23:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 30 fb 74 |..........4740.t| 2024-05-17T15:23:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 72 ea 6c 05 0a 00 00 00 01 01 04 00 00 00 |Gfr.l...........| 2024-05-17T15:23:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_bef15777-4cd7-47fb-8829-12dabc805b90 2024-05-17T15:23:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 39 38 36 89 74 |..........4986.t| 2024-05-17T15:23:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 3c fc eb 10 0a 00 00 00 01 01 04 00 00 00 |Gf<.............| 2024-05-17T15:23:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_c50f08e5-ef2e-4cc9-afd7-fb18dd3f03dc 2024-05-17T15:23:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 39 38 36 89 74 |..........4986.t| 2024-05-17T15:23:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 89 05 ec 10 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_ceaa7752-4708-4453-aad7-8c9fe4b5ad62 2024-05-17T15:23:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 39 38 36 89 74 |..........4986.t| 2024-05-17T15:23:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 90 0e ec 10 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_d0b80867-2a3b-4947-ae2d-30cd6b4ea8cc 2024-05-17T15:23:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 35 31 36 39 e7 74 |..........5169.t| 2024-05-17T15:23:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 e2 89 38 04 0a 00 00 00 01 01 04 00 00 00 |Gf..8...........| 2024-05-17T15:23:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_d95d4f33-deca-446d-88ca-2fd4ef56f554 2024-05-17T15:23:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 30 97 74 |..........4740.t| 2024-05-17T15:23:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 ce fc ac 11 0a 00 00 00 01 01 04 00 00 00 |Gf..............| 2024-05-17T15:23:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_e3330572-d02a-4104-a14f-d83973929a2c 2024-05-17T15:23:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 30 3e 74 |..........4740>t| 2024-05-17T15:23:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 53 12 8d 0e 0a 00 00 00 01 01 04 00 00 00 |GfS.............| 2024-05-17T15:23:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_f990d936-a771-483b-8728-fd1503201136 2024-05-17T15:23:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 37 34 30 a4 73 |..........4740.s| 2024-05-17T15:23:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 57 ca d2 30 0a 00 00 00 01 01 04 00 00 00 |GfW..0..........| 2024-05-17T15:23:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image_map_fb0ce489-3f67-44e1-971c-03b1005074a7 2024-05-17T15:23:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 30 39 fb 74 |..........4809.t| 2024-05-17T15:23:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 47 66 59 05 6d 05 0a 00 00 00 01 01 04 00 00 00 |GfY.m...........| 2024-05-17T15:23:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 00 00 00 00 |....| 2024-05-17T15:23:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:instance_4740 2024-05-17T15:23:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (0 bytes) : 2024-05-17T15:23:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:instance_4809 2024-05-17T15:23:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (0 bytes) : 2024-05-17T15:23:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:instance_4986 2024-05-17T15:23:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (0 bytes) : 2024-05-17T15:23:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:instance_5169 2024-05-17T15:23:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (0 bytes) : 2024-05-17T15:23:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-05-17T15:23:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror ns1 images 2024-05-17T15:23:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror ns1 images' 2024-05-17T15:23:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 ls -l 2024-05-17T15:23:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirrorns1 mirror pool info 2024-05-17T15:23:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirrorns1 mirror pool info' 2024-05-17T15:23:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool info 2024-05-17T15:23:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mode: image 2024-05-17T15:23:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirrorns1 mirror pool status 2024-05-17T15:23:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirrorns1 mirror pool status' 2024-05-17T15:23:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T15:23:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool status --verbose 2024-05-17T15:23:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:health: OK 2024-05-17T15:23:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:daemon health: OK 2024-05-17T15:23:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image health: OK 2024-05-17T15:23:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:images: 0 total 2024-05-17T15:23:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:DAEMONS 2024-05-17T15:23:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:service 4737: 2024-05-17T15:23:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: instance_id: 4740 2024-05-17T15:23:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: client_id: mirror.2 2024-05-17T15:23:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: hostname: smithi195 2024-05-17T15:23:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: version: 19.0.0-3735-gee59d86a 2024-05-17T15:23:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: leader: true 2024-05-17T15:23:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: health: OK 2024-05-17T15:23:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:service 4806: 2024-05-17T15:23:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: instance_id: 4809 2024-05-17T15:23:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: client_id: mirror.0 2024-05-17T15:23:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: hostname: smithi195 2024-05-17T15:23:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: version: 19.0.0-3735-gee59d86a 2024-05-17T15:23:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: leader: false 2024-05-17T15:23:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: health: OK 2024-05-17T15:23:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:service 4983: 2024-05-17T15:23:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: instance_id: 4986 2024-05-17T15:23:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: client_id: mirror.1 2024-05-17T15:23:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: hostname: smithi195 2024-05-17T15:23:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: version: 19.0.0-3735-gee59d86a 2024-05-17T15:23:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: leader: false 2024-05-17T15:23:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: health: OK 2024-05-17T15:23:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:service 5166: 2024-05-17T15:23:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: instance_id: 5169 2024-05-17T15:23:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: client_id: mirror.3 2024-05-17T15:23:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: hostname: smithi195 2024-05-17T15:23:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: version: 19.0.0-3735-gee59d86a 2024-05-17T15:23:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: leader: false 2024-05-17T15:23:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: health: OK 2024-05-17T15:23:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:IMAGES 2024-05-17T15:23:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns1 ls 2024-05-17T15:23:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror ns1 rbd_mirroring omap vals 2024-05-17T15:23:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror ns1 rbd_mirroring omap vals' 2024-05-17T15:23:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirroring 2024-05-17T15:23:25.114 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:mirror_mode 2024-05-17T15:23:25.114 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (4 bytes) : 2024-05-17T15:23:25.114 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 00 00 00 |....| 2024-05-17T15:23:25.114 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000004 2024-05-17T15:23:25.114 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.114 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:mirror_uuid 2024-05-17T15:23:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 65 34 30 62 39 62 30 65 2d 39 66 61 37 2d 34 64 |e40b9b0e-9fa7-4d| 2024-05-17T15:23:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 30 39 2d 39 38 39 36 2d 35 31 31 34 37 64 64 38 |09-9896-51147dd8| 2024-05-17T15:23:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 34 65 62 35 |4eb5| 2024-05-17T15:23:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror ns1 rbd_mirror_leader omap vals' 2024-05-17T15:23:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror ns1 rbd_mirror_leader omap vals 2024-05-17T15:23:25.117 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2024-05-17T15:23:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:instance_4740 2024-05-17T15:23:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (0 bytes) : 2024-05-17T15:23:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:instance_4986 2024-05-17T15:23:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (0 bytes) : 2024-05-17T15:23:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:instance_5169 2024-05-17T15:23:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (0 bytes) : 2024-05-17T15:23:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-05-17T15:23:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror ns2 images' 2024-05-17T15:23:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror ns2 images 2024-05-17T15:23:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 ls -l 2024-05-17T15:23:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirrorns2 mirror pool info 2024-05-17T15:23:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirrorns2 mirror pool info' 2024-05-17T15:23:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool info 2024-05-17T15:23:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mode: image 2024-05-17T15:23:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirrorns2 mirror pool status 2024-05-17T15:23:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirrorns2 mirror pool status' 2024-05-17T15:23:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T15:23:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool status --verbose 2024-05-17T15:23:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:health: OK 2024-05-17T15:23:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:daemon health: OK 2024-05-17T15:23:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image health: OK 2024-05-17T15:23:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:images: 0 total 2024-05-17T15:23:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:DAEMONS 2024-05-17T15:23:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:service 4737: 2024-05-17T15:23:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: instance_id: 4740 2024-05-17T15:23:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: client_id: mirror.2 2024-05-17T15:23:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: hostname: smithi195 2024-05-17T15:23:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: version: 19.0.0-3735-gee59d86a 2024-05-17T15:23:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: leader: true 2024-05-17T15:23:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: health: OK 2024-05-17T15:23:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:service 4806: 2024-05-17T15:23:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: instance_id: 4809 2024-05-17T15:23:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: client_id: mirror.0 2024-05-17T15:23:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: hostname: smithi195 2024-05-17T15:23:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: version: 19.0.0-3735-gee59d86a 2024-05-17T15:23:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: leader: false 2024-05-17T15:23:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: health: OK 2024-05-17T15:23:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:service 4983: 2024-05-17T15:23:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: instance_id: 4986 2024-05-17T15:23:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: client_id: mirror.1 2024-05-17T15:23:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: hostname: smithi195 2024-05-17T15:23:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: version: 19.0.0-3735-gee59d86a 2024-05-17T15:23:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: leader: false 2024-05-17T15:23:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: health: OK 2024-05-17T15:23:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:service 5166: 2024-05-17T15:23:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: instance_id: 5169 2024-05-17T15:23:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: client_id: mirror.3 2024-05-17T15:23:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: hostname: smithi195 2024-05-17T15:23:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: version: 19.0.0-3735-gee59d86a 2024-05-17T15:23:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: leader: false 2024-05-17T15:23:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: health: OK 2024-05-17T15:23:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:IMAGES 2024-05-17T15:23:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns2 ls 2024-05-17T15:23:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror ns2 rbd_mirroring omap vals 2024-05-17T15:23:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror ns2 rbd_mirroring omap vals' 2024-05-17T15:23:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirroring 2024-05-17T15:23:25.335 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:mirror_mode 2024-05-17T15:23:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (4 bytes) : 2024-05-17T15:23:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 00 00 00 |....| 2024-05-17T15:23:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000004 2024-05-17T15:23:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:mirror_uuid 2024-05-17T15:23:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 31 38 31 65 37 66 37 38 2d 32 66 32 39 2d 34 32 |181e7f78-2f29-42| 2024-05-17T15:23:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 62 39 2d 38 66 63 30 2d 33 64 62 66 33 65 36 62 |b9-8fc0-3dbf3e6b| 2024-05-17T15:23:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 36 39 35 64 |695d| 2024-05-17T15:23:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror ns2 rbd_mirror_leader omap vals' 2024-05-17T15:23:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror ns2 rbd_mirror_leader omap vals 2024-05-17T15:23:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2024-05-17T15:23:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:instance_4740 2024-05-17T15:23:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (0 bytes) : 2024-05-17T15:23:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:instance_4986 2024-05-17T15:23:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (0 bytes) : 2024-05-17T15:23:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:instance_5169 2024-05-17T15:23:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (0 bytes) : 2024-05-17T15:23:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2024-05-17T15:23:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-05-17T15:23:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror_parent images' 2024-05-17T15:23:25.374 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror_parent images 2024-05-17T15:23:25.374 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace '' ls -l 2024-05-17T15:23:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror_parent mirror pool info 2024-05-17T15:23:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror_parent mirror pool info' 2024-05-17T15:23:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace '' mirror pool info 2024-05-17T15:23:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mode: image 2024-05-17T15:23:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Site Name: cluster1 2024-05-17T15:23:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Peer Sites: 2024-05-17T15:23:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:UUID: 0a5f21f5-898c-4c37-b257-500dcef76f47 2024-05-17T15:23:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Name: cluster2 2024-05-17T15:23:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Mirror UUID: 6ece31da-a3ee-4d69-9e88-9ee3a2d3dd74 2024-05-17T15:23:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Direction: rx-tx 2024-05-17T15:23:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:Client: client.mirror 2024-05-17T15:23:25.446 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror_parent mirror pool status 2024-05-17T15:23:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror_parent mirror pool status' 2024-05-17T15:23:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T15:23:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace '' mirror pool status --verbose 2024-05-17T15:23:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:health: WARNING 2024-05-17T15:23:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:daemon health: WARNING 2024-05-17T15:23:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:image health: OK 2024-05-17T15:23:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:images: 0 total 2024-05-17T15:23:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:DAEMONS 2024-05-17T15:23:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:service 4737: 2024-05-17T15:23:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: instance_id: 4743 2024-05-17T15:23:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: client_id: mirror.2 2024-05-17T15:23:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: hostname: smithi195 2024-05-17T15:23:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: version: 19.0.0-3735-gee59d86a 2024-05-17T15:23:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: leader: false 2024-05-17T15:23:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: health: OK 2024-05-17T15:23:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:service 4806: 2024-05-17T15:23:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: instance_id: 4811 2024-05-17T15:23:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: client_id: mirror.0 2024-05-17T15:23:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: hostname: smithi195 2024-05-17T15:23:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: version: 19.0.0-3735-gee59d86a 2024-05-17T15:23:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: leader: false 2024-05-17T15:23:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: health: OK 2024-05-17T15:23:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:service 4983: 2024-05-17T15:23:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: instance_id: 4989 2024-05-17T15:23:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: client_id: mirror.1 2024-05-17T15:23:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: hostname: smithi195 2024-05-17T15:23:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: version: 19.0.0-3735-gee59d86a 2024-05-17T15:23:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: leader: false 2024-05-17T15:23:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: health: OK 2024-05-17T15:23:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:service 5166: 2024-05-17T15:23:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: instance_id: 5172 2024-05-17T15:23:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: client_id: mirror.3 2024-05-17T15:23:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: hostname: smithi195 2024-05-17T15:23:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: version: 19.0.0-3735-gee59d86a 2024-05-17T15:23:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: leader: false 2024-05-17T15:23:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: health: OK 2024-05-17T15:23:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:IMAGES 2024-05-17T15:23:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace '' ls 2024-05-17T15:23:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror_parent rbd_mirroring omap vals 2024-05-17T15:23:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror_parent rbd_mirroring omap vals' 2024-05-17T15:23:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace '' listomapvals rbd_mirroring 2024-05-17T15:23:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:mirror_mode 2024-05-17T15:23:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (4 bytes) : 2024-05-17T15:23:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 01 00 00 00 |....| 2024-05-17T15:23:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000004 2024-05-17T15:23:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:mirror_peer_0a5f21f5-898c-4c37-b257-500dcef76f47 2024-05-17T15:23:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (132 bytes) : 2024-05-17T15:23:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 30 61 35 66 32 31 |..~...$...0a5f21| 2024-05-17T15:23:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 66 35 2d 38 39 38 63 2d 34 63 33 37 2d 62 32 35 |f5-898c-4c37-b25| 2024-05-17T15:23:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 37 2d 35 30 30 64 63 65 66 37 36 66 34 37 08 00 |7-500dcef76f47..| 2024-05-17T15:23:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000030 00 00 63 6c 75 73 74 65 72 32 0d 00 00 00 63 6c |..cluster2....cl| 2024-05-17T15:23:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2024-05-17T15:23:25.556 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000050 ff ff ff 02 24 00 00 00 36 65 63 65 33 31 64 61 |....$...6ece31da| 2024-05-17T15:23:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000060 2d 61 33 65 65 2d 34 64 36 39 2d 39 65 38 38 2d |-a3ee-4d69-9e88-| 2024-05-17T15:23:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000070 39 65 65 33 61 32 64 33 64 64 37 34 69 76 47 66 |9ee3a2d3dd74ivGf| 2024-05-17T15:23:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000080 f0 2a be 23 |.*.#| 2024-05-17T15:23:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000084 2024-05-17T15:23:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:mirror_uuid 2024-05-17T15:23:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (36 bytes) : 2024-05-17T15:23:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000000 36 66 61 63 35 33 39 39 2d 35 64 65 38 2d 34 33 |6fac5399-5de8-43| 2024-05-17T15:23:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000010 35 34 2d 38 32 35 34 2d 38 37 36 63 61 39 65 61 |54-8254-876ca9ea| 2024-05-17T15:23:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000020 66 34 62 39 |f4b9| 2024-05-17T15:23:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:00000024 2024-05-17T15:23:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror_parent rbd_mirror_leader omap vals' 2024-05-17T15:23:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror_parent rbd_mirror_leader omap vals 2024-05-17T15:23:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace '' listomapvals rbd_mirror_leader 2024-05-17T15:23:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:instance_4743 2024-05-17T15:23:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (0 bytes) : 2024-05-17T15:23:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:instance_4811 2024-05-17T15:23:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (0 bytes) : 2024-05-17T15:23:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:instance_4989 2024-05-17T15:23:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (0 bytes) : 2024-05-17T15:23:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:instance_5172 2024-05-17T15:23:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:value (0 bytes) : 2024-05-17T15:23:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-05-17T15:23:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror_parent ns1 images 2024-05-17T15:23:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror_parent ns1 images' 2024-05-17T15:23:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls -l 2024-05-17T15:23:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: namespace 'ns1' does not exist. 2024-05-17T15:23:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: listing images failed: (2) No such file or directory 2024-05-17T15:23:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror_parentns1 mirror pool info 2024-05-17T15:23:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror_parentns1 mirror pool info' 2024-05-17T15:23:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool info 2024-05-17T15:23:25.664 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: namespace 'ns1' does not exist. 2024-05-17T15:23:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror_parentns1 mirror pool status 2024-05-17T15:23:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror_parentns1 mirror pool status' 2024-05-17T15:23:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T15:23:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool status --verbose 2024-05-17T15:23:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: namespace 'ns1' does not exist. 2024-05-17T15:23:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls 2024-05-17T15:23:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror_parent ns1 rbd_mirroring omap vals 2024-05-17T15:23:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror_parent ns1 rbd_mirroring omap vals' 2024-05-17T15:23:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2024-05-17T15:23:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2024-05-17T15:23:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror_parent ns1 rbd_mirror_leader omap vals' 2024-05-17T15:23:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror_parent ns1 rbd_mirror_leader omap vals 2024-05-17T15:23:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirror_leader 2024-05-17T15:23:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:error getting omap keys mirror_parent/rbd_mirror_leader: (2) No such file or directory 2024-05-17T15:23:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-05-17T15:23:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror_parent ns2 images 2024-05-17T15:23:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror_parent ns2 images' 2024-05-17T15:23:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls -l 2024-05-17T15:23:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: namespace 'ns2' does not exist. 2024-05-17T15:23:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: listing images failed: (2) No such file or directory 2024-05-17T15:23:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror_parentns2 mirror pool info 2024-05-17T15:23:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror_parentns2 mirror pool info' 2024-05-17T15:23:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool info 2024-05-17T15:23:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: namespace 'ns2' does not exist. 2024-05-17T15:23:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror_parentns2 mirror pool status 2024-05-17T15:23:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror_parentns2 mirror pool status' 2024-05-17T15:23:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T15:23:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool status --verbose 2024-05-17T15:23:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: namespace 'ns2' does not exist. 2024-05-17T15:23:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls 2024-05-17T15:23:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror_parent ns2 rbd_mirroring omap vals 2024-05-17T15:23:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror_parent ns2 rbd_mirroring omap vals' 2024-05-17T15:23:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2024-05-17T15:23:25.986 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2024-05-17T15:23:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 mirror_parent ns2 rbd_mirror_leader omap vals' 2024-05-17T15:23:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 mirror_parent ns2 rbd_mirror_leader omap vals 2024-05-17T15:23:25.989 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirror_leader 2024-05-17T15:23:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:error getting omap keys mirror_parent/rbd_mirror_leader: (2) No such file or directory 2024-05-17T15:23:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:23:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:23:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2024-05-17T15:23:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 status 2024-05-17T15:23:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 status' 2024-05-17T15:23:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T15:23:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ ceph --cluster cluster2 -s 2024-05-17T15:28:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:2024-05-17T15:28:26.297+0000 7f77a75e8640 0 monclient(hunting): authenticate timed out after 300 2024-05-17T15:28:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-05-17T15:28:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T15:28:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ ceph --cluster cluster2 service dump 2024-05-17T15:29:48.157 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster1.client.mirror.0 2024-05-17T15:29:48.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 6 2024-05-17T15:29:48.159 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2024-05-17T15:29:48.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:*** Caught signal (Aborted) ** 2024-05-17T15:29:48.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: in thread 7fc689e33640 thread_name:rbd-mirror 2024-05-17T15:29:48.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev) 2024-05-17T15:29:48.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc68983e6f0] 2024-05-17T15:29:48.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 2: /lib64/libc.so.6(+0x8679a) [0x7fc68988679a] 2024-05-17T15:29:48.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 3: pthread_cond_wait() 2024-05-17T15:29:48.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fc689cd56b0] 2024-05-17T15:29:48.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 5: (rbd::mirror::LeaderWatcher::shut_down()+0xeb) [0x5557de0e673b] 2024-05-17T15:29:48.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x91) [0x5557de0c11a1] 2024-05-17T15:29:48.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x5557de0b4550] 2024-05-17T15:29:48.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 8: rbd-mirror(+0x98bf58) [0x5557de6cff58] 2024-05-17T15:29:48.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 9: rbd-mirror(+0x98bf40) [0x5557de6cff40] 2024-05-17T15:29:48.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xca) [0x5557de0b2a7a] 2024-05-17T15:29:48.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 11: main() 2024-05-17T15:29:48.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 12: /lib64/libc.so.6(+0x29590) [0x7fc689829590] 2024-05-17T15:29:48.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 13: __libc_start_main() 2024-05-17T15:29:48.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 14: _start() 2024-05-17T15:29:48.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:2024-05-17T15:29:48.159+0000 7fc689e33640 -1 *** Caught signal (Aborted) ** 2024-05-17T15:29:48.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: in thread 7fc689e33640 thread_name:rbd-mirror 2024-05-17T15:29:48.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 2024-05-17T15:29:48.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev) 2024-05-17T15:29:48.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc68983e6f0] 2024-05-17T15:29:48.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 2: /lib64/libc.so.6(+0x8679a) [0x7fc68988679a] 2024-05-17T15:29:48.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 3: pthread_cond_wait() 2024-05-17T15:29:48.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fc689cd56b0] 2024-05-17T15:29:48.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 5: (rbd::mirror::LeaderWatcher::shut_down()+0xeb) [0x5557de0e673b] 2024-05-17T15:29:48.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x91) [0x5557de0c11a1] 2024-05-17T15:29:48.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x5557de0b4550] 2024-05-17T15:29:48.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 8: rbd-mirror(+0x98bf58) [0x5557de6cff58] 2024-05-17T15:29:48.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 9: rbd-mirror(+0x98bf40) [0x5557de6cff40] 2024-05-17T15:29:48.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xca) [0x5557de0b2a7a] 2024-05-17T15:29:48.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 11: main() 2024-05-17T15:29:48.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 12: /lib64/libc.so.6(+0x29590) [0x7fc689829590] 2024-05-17T15:29:48.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 13: __libc_start_main() 2024-05-17T15:29:48.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 14: _start() 2024-05-17T15:29:48.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-17T15:29:48.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 2024-05-17T15:29:48.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 0> 2024-05-17T15:29:48.159+0000 7fc689e33640 -1 *** Caught signal (Aborted) ** 2024-05-17T15:29:48.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: in thread 7fc689e33640 thread_name:rbd-mirror 2024-05-17T15:29:48.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 2024-05-17T15:29:48.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev) 2024-05-17T15:29:48.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc68983e6f0] 2024-05-17T15:29:48.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 2: /lib64/libc.so.6(+0x8679a) [0x7fc68988679a] 2024-05-17T15:29:48.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 3: pthread_cond_wait() 2024-05-17T15:29:48.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fc689cd56b0] 2024-05-17T15:29:48.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 5: (rbd::mirror::LeaderWatcher::shut_down()+0xeb) [0x5557de0e673b] 2024-05-17T15:29:48.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x91) [0x5557de0c11a1] 2024-05-17T15:29:48.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x5557de0b4550] 2024-05-17T15:29:48.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 8: rbd-mirror(+0x98bf58) [0x5557de6cff58] 2024-05-17T15:29:48.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 9: rbd-mirror(+0x98bf40) [0x5557de6cff40] 2024-05-17T15:29:48.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xca) [0x5557de0b2a7a] 2024-05-17T15:29:48.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 11: main() 2024-05-17T15:29:48.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 12: /lib64/libc.so.6(+0x29590) [0x7fc689829590] 2024-05-17T15:29:48.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 13: __libc_start_main() 2024-05-17T15:29:48.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 14: _start() 2024-05-17T15:29:48.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-17T15:29:48.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 2024-05-17T15:29:48.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: -9999> 2024-05-17T15:29:48.159+0000 7fc689e33640 -1 *** Caught signal (Aborted) ** 2024-05-17T15:29:48.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: in thread 7fc689e33640 thread_name:rbd-mirror 2024-05-17T15:29:48.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 2024-05-17T15:29:48.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev) 2024-05-17T15:29:48.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc68983e6f0] 2024-05-17T15:29:48.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 2: /lib64/libc.so.6(+0x8679a) [0x7fc68988679a] 2024-05-17T15:29:48.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 3: pthread_cond_wait() 2024-05-17T15:29:48.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fc689cd56b0] 2024-05-17T15:29:48.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 5: (rbd::mirror::LeaderWatcher::shut_down()+0xeb) [0x5557de0e673b] 2024-05-17T15:29:48.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 6: (rbd::mirror::PoolReplayer::shut_down()+0x91) [0x5557de0c11a1] 2024-05-17T15:29:48.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 7: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x5557de0b4550] 2024-05-17T15:29:48.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 8: rbd-mirror(+0x98bf58) [0x5557de6cff58] 2024-05-17T15:29:48.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 9: rbd-mirror(+0x98bf40) [0x5557de6cff40] 2024-05-17T15:29:48.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 10: (rbd::mirror::Mirror::~Mirror()+0xca) [0x5557de0b2a7a] 2024-05-17T15:29:48.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 11: main() 2024-05-17T15:29:48.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 12: /lib64/libc.so.6(+0x29590) [0x7fc689829590] 2024-05-17T15:29:48.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 13: __libc_start_main() 2024-05-17T15:29:48.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 14: _start() 2024-05-17T15:29:48.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-17T15:29:48.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr: 2024-05-17T15:29:48.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi195.stderr:daemon-helper: command crashed with signal 6 2024-05-17T15:29:53.368 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.rbd_mirror.[cluster1] failed 2024-05-17T15:29:53.369 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-05-17T15:29:53.872 INFO:tasks.ceph.cluster1.osd.0:Sent signal 15 2024-05-17T15:29:53.873 INFO:tasks.ceph.cluster1.osd.1:Sent signal 15 2024-05-17T15:29:53.873 INFO:tasks.ceph.cluster1.osd.2:Sent signal 15 2024-05-17T15:29:53.873 INFO:tasks.ceph.cluster1.mon.a:Sent signal 15 2024-05-17T15:29:53.874 INFO:tasks.ceph.cluster1.mgr.x:Sent signal 15 2024-05-17T15:29:53.874 INFO:tasks.ceph.cluster1.osd.0.smithi134.stderr:2024-05-17T15:29:53.871+0000 7fd85b08a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 (PID: 32710) UID: 0 2024-05-17T15:29:53.874 INFO:tasks.ceph.cluster1.osd.0.smithi134.stderr:2024-05-17T15:29:53.871+0000 7fd85b08a640 -1 osd.0 244 *** Got signal Terminated *** 2024-05-17T15:29:53.874 INFO:tasks.ceph.cluster1.osd.0.smithi134.stderr:2024-05-17T15:29:53.871+0000 7fd85b08a640 -1 osd.0 244 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-17T15:29:53.875 INFO:tasks.ceph.cluster1.osd.1.smithi134.stderr:2024-05-17T15:29:53.872+0000 7f346b15d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 (PID: 32713) UID: 0 2024-05-17T15:29:53.875 INFO:tasks.ceph.cluster1.osd.1.smithi134.stderr:2024-05-17T15:29:53.872+0000 7f346b15d640 -1 osd.1 244 *** Got signal Terminated *** 2024-05-17T15:29:53.875 INFO:tasks.ceph.cluster1.osd.1.smithi134.stderr:2024-05-17T15:29:53.872+0000 7f346b15d640 -1 osd.1 244 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-17T15:29:53.875 INFO:tasks.ceph.cluster1.osd.2.smithi134.stderr:2024-05-17T15:29:53.872+0000 7f436d3f4640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 (PID: 32715) UID: 0 2024-05-17T15:29:53.875 INFO:tasks.ceph.cluster1.osd.2.smithi134.stderr:2024-05-17T15:29:53.872+0000 7f436d3f4640 -1 osd.2 244 *** Got signal Terminated *** 2024-05-17T15:29:53.875 INFO:tasks.ceph.cluster1.osd.2.smithi134.stderr:2024-05-17T15:29:53.872+0000 7f436d3f4640 -1 osd.2 244 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-17T15:29:53.876 INFO:tasks.ceph.cluster1.mon.a.smithi134.stderr:2024-05-17T15:29:53.872+0000 7f0785927640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster1 -i a (PID: 32377) UID: 0 2024-05-17T15:29:53.876 INFO:tasks.ceph.cluster1.mon.a.smithi134.stderr:2024-05-17T15:29:53.872+0000 7f0785927640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-17T15:29:53.876 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T15:29:53.873+0000 7fd16912c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster1 -i x (PID: 32381) UID: 0 2024-05-17T15:29:53.876 INFO:tasks.ceph.cluster1.mgr.x.smithi134.stderr:2024-05-17T15:29:53.873+0000 7fd16912c640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-05-17T15:29:59.166 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster2.client.mirror.0 2024-05-17T15:29:59.166 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 6 2024-05-17T15:29:59.167 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2024-05-17T15:29:59.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr:*** Caught signal (Aborted) ** 2024-05-17T15:29:59.168 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: in thread 7f15dca67640 thread_name:rbd-mirror 2024-05-17T15:29:59.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev) 2024-05-17T15:29:59.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f15dce3e6f0] 2024-05-17T15:29:59.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 2: /lib64/libc.so.6(+0x8679a) [0x7f15dce8679a] 2024-05-17T15:29:59.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 3: pthread_cond_wait() 2024-05-17T15:29:59.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f15dd2d56b0] 2024-05-17T15:29:59.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 5: /lib64/librados.so.2(+0x14553b) [0x7f15deb5753b] 2024-05-17T15:29:59.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x5566acd08768] 2024-05-17T15:29:59.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x5566acd0a5e8] 2024-05-17T15:29:59.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x5566acce9d44] 2024-05-17T15:29:59.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 9: main() 2024-05-17T15:29:59.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 10: /lib64/libc.so.6(+0x29590) [0x7f15dce29590] 2024-05-17T15:29:59.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 11: __libc_start_main() 2024-05-17T15:29:59.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 12: _start() 2024-05-17T15:29:59.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr:2024-05-17T15:29:59.167+0000 7f15dca67640 -1 *** Caught signal (Aborted) ** 2024-05-17T15:29:59.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: in thread 7f15dca67640 thread_name:rbd-mirror 2024-05-17T15:29:59.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 2024-05-17T15:29:59.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev) 2024-05-17T15:29:59.170 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f15dce3e6f0] 2024-05-17T15:29:59.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 2: /lib64/libc.so.6(+0x8679a) [0x7f15dce8679a] 2024-05-17T15:29:59.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 3: pthread_cond_wait() 2024-05-17T15:29:59.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f15dd2d56b0] 2024-05-17T15:29:59.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 5: /lib64/librados.so.2(+0x14553b) [0x7f15deb5753b] 2024-05-17T15:29:59.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x5566acd08768] 2024-05-17T15:29:59.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x5566acd0a5e8] 2024-05-17T15:29:59.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x5566acce9d44] 2024-05-17T15:29:59.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 9: main() 2024-05-17T15:29:59.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 10: /lib64/libc.so.6(+0x29590) [0x7f15dce29590] 2024-05-17T15:29:59.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 11: __libc_start_main() 2024-05-17T15:29:59.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 12: _start() 2024-05-17T15:29:59.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-17T15:29:59.171 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 2024-05-17T15:29:59.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: -1963> 2024-05-17T15:19:50.015+0000 7f15d9a11640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 56208) UID: 1000 2024-05-17T15:29:59.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 0> 2024-05-17T15:29:59.167+0000 7f15dca67640 -1 *** Caught signal (Aborted) ** 2024-05-17T15:29:59.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: in thread 7f15dca67640 thread_name:rbd-mirror 2024-05-17T15:29:59.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 2024-05-17T15:29:59.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev) 2024-05-17T15:29:59.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f15dce3e6f0] 2024-05-17T15:29:59.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 2: /lib64/libc.so.6(+0x8679a) [0x7f15dce8679a] 2024-05-17T15:29:59.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 3: pthread_cond_wait() 2024-05-17T15:29:59.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f15dd2d56b0] 2024-05-17T15:29:59.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 5: /lib64/librados.so.2(+0x14553b) [0x7f15deb5753b] 2024-05-17T15:29:59.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x5566acd08768] 2024-05-17T15:29:59.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x5566acd0a5e8] 2024-05-17T15:29:59.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x5566acce9d44] 2024-05-17T15:29:59.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 9: main() 2024-05-17T15:29:59.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 10: /lib64/libc.so.6(+0x29590) [0x7f15dce29590] 2024-05-17T15:29:59.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 11: __libc_start_main() 2024-05-17T15:29:59.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 12: _start() 2024-05-17T15:29:59.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-17T15:29:59.184 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 2024-05-17T15:29:59.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: -9043> 2024-05-17T15:19:50.015+0000 7f15d9a11640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 56208) UID: 1000 2024-05-17T15:29:59.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: -9042> 2024-05-17T15:29:59.167+0000 7f15dca67640 -1 *** Caught signal (Aborted) ** 2024-05-17T15:29:59.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: in thread 7f15dca67640 thread_name:rbd-mirror 2024-05-17T15:29:59.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 2024-05-17T15:29:59.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: ceph version 19.0.0-3735-gee59d86a (ee59d86aad04089481e55c1a15aa5871339935b4) squid (dev) 2024-05-17T15:29:59.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f15dce3e6f0] 2024-05-17T15:29:59.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 2: /lib64/libc.so.6(+0x8679a) [0x7f15dce8679a] 2024-05-17T15:29:59.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 3: pthread_cond_wait() 2024-05-17T15:29:59.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f15dd2d56b0] 2024-05-17T15:29:59.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 5: /lib64/librados.so.2(+0x14553b) [0x7f15deb5753b] 2024-05-17T15:29:59.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 6: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x38) [0x5566acd08768] 2024-05-17T15:29:59.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 7: (rbd::mirror::ClusterWatcher::refresh_pools()+0x78) [0x5566acd0a5e8] 2024-05-17T15:29:59.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 8: (rbd::mirror::Mirror::run()+0xb4) [0x5566acce9d44] 2024-05-17T15:29:59.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 9: main() 2024-05-17T15:29:59.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 10: /lib64/libc.so.6(+0x29590) [0x7f15dce29590] 2024-05-17T15:29:59.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 11: __libc_start_main() 2024-05-17T15:29:59.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 12: _start() 2024-05-17T15:29:59.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-17T15:29:59.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr: 2024-05-17T15:29:59.569 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi195.stderr:daemon-helper: command crashed with signal 6 2024-05-17T15:33:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:2024-05-17T15:33:26.385+0000 7f3b2564c640 0 monclient(hunting): authenticate timed out after 300 2024-05-17T15:33:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-05-17T15:33:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T15:33:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ ceph --cluster cluster2 service status 2024-05-17T15:38:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:2024-05-17T15:38:26.471+0000 7f9f76895640 0 monclient(hunting): authenticate timed out after 300 2024-05-17T15:38:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-05-17T15:38:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2024-05-17T15:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-05-17T15:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror images' 2024-05-17T15:38:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror images 2024-05-17T15:38:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' ls -l 2024-05-17T15:43:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T15:43:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: listing images failed: (110) Connection timed out 2024-05-17T15:43:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:43:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:43:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror mirror pool info 2024-05-17T15:43:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror mirror pool info' 2024-05-17T15:43:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' mirror pool info 2024-05-17T15:48:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T15:48:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:48:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:48:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror mirror pool status 2024-05-17T15:48:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror mirror pool status' 2024-05-17T15:48:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T15:48:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' mirror pool status --verbose 2024-05-17T15:53:26.578 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T15:53:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T15:53:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T15:53:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster2 -p mirror --namespace '' ls 2024-05-17T15:58:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror rbd_mirroring omap vals 2024-05-17T15:58:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror rbd_mirroring omap vals' 2024-05-17T15:58:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirroring 2024-05-17T16:03:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-05-17T16:03:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror rbd_mirror_leader omap vals 2024-05-17T16:03:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror rbd_mirror_leader omap vals' 2024-05-17T16:03:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirror_leader 2024-05-17T16:08:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-05-17T16:08:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T16:08:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T16:08:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-05-17T16:08:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror ns1 images' 2024-05-17T16:08:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror ns1 images 2024-05-17T16:08:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 ls -l 2024-05-17T16:13:26.706 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T16:13:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: listing images failed: (110) Connection timed out 2024-05-17T16:13:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T16:13:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T16:13:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirrorns1 mirror pool info 2024-05-17T16:13:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirrorns1 mirror pool info' 2024-05-17T16:13:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool info 2024-05-17T16:18:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T16:18:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T16:18:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T16:18:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirrorns1 mirror pool status 2024-05-17T16:18:26.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirrorns1 mirror pool status' 2024-05-17T16:18:26.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T16:18:26.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool status --verbose 2024-05-17T16:23:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T16:23:26.776 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T16:23:26.777 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T16:23:26.777 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns1 ls 2024-05-17T16:28:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror ns1 rbd_mirroring omap vals 2024-05-17T16:28:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror ns1 rbd_mirroring omap vals' 2024-05-17T16:28:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirroring 2024-05-17T16:33:26.840 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-05-17T16:33:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror ns1 rbd_mirror_leader omap vals 2024-05-17T16:33:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror ns1 rbd_mirror_leader omap vals' 2024-05-17T16:33:26.842 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2024-05-17T16:38:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-05-17T16:38:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T16:38:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T16:38:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-05-17T16:38:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror ns2 images 2024-05-17T16:38:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror ns2 images' 2024-05-17T16:38:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 ls -l 2024-05-17T16:43:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T16:43:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: listing images failed: (110) Connection timed out 2024-05-17T16:43:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T16:43:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T16:43:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirrorns2 mirror pool info 2024-05-17T16:43:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirrorns2 mirror pool info' 2024-05-17T16:43:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool info 2024-05-17T16:48:26.939 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T16:48:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T16:48:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T16:48:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirrorns2 mirror pool status' 2024-05-17T16:48:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirrorns2 mirror pool status 2024-05-17T16:48:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T16:48:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool status --verbose 2024-05-17T16:53:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T16:53:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T16:53:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T16:53:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns2 ls 2024-05-17T16:58:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror ns2 rbd_mirroring omap vals 2024-05-17T16:58:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror ns2 rbd_mirroring omap vals' 2024-05-17T16:58:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirroring 2024-05-17T17:03:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-05-17T17:03:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror ns2 rbd_mirror_leader omap vals 2024-05-17T17:03:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror ns2 rbd_mirror_leader omap vals' 2024-05-17T17:03:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2024-05-17T17:08:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-05-17T17:08:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T17:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T17:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2024-05-17T17:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-05-17T17:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror_parent images 2024-05-17T17:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror_parent images' 2024-05-17T17:08:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' ls -l 2024-05-17T17:13:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T17:13:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: listing images failed: (110) Connection timed out 2024-05-17T17:13:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T17:13:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T17:13:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror_parent mirror pool info' 2024-05-17T17:13:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror_parent mirror pool info 2024-05-17T17:13:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool info 2024-05-17T17:18:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T17:18:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T17:18:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T17:18:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror_parent mirror pool status 2024-05-17T17:18:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror_parent mirror pool status' 2024-05-17T17:18:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T17:18:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool status --verbose 2024-05-17T17:23:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T17:23:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T17:23:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T17:23:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace '' ls 2024-05-17T17:28:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror_parent rbd_mirroring omap vals 2024-05-17T17:28:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror_parent rbd_mirroring omap vals' 2024-05-17T17:28:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirroring 2024-05-17T17:33:27.236 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-05-17T17:33:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror_parent rbd_mirror_leader omap vals 2024-05-17T17:33:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror_parent rbd_mirror_leader omap vals' 2024-05-17T17:33:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirror_leader 2024-05-17T17:38:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-05-17T17:38:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T17:38:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T17:38:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-05-17T17:38:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror_parent ns1 images 2024-05-17T17:38:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror_parent ns1 images' 2024-05-17T17:38:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls -l 2024-05-17T17:43:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T17:43:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: listing images failed: (110) Connection timed out 2024-05-17T17:43:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T17:43:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T17:43:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror_parentns1 mirror pool info 2024-05-17T17:43:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror_parentns1 mirror pool info' 2024-05-17T17:43:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool info 2024-05-17T17:48:27.332 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T17:48:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T17:48:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror_parentns1 mirror pool status 2024-05-17T17:48:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T17:48:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror_parentns1 mirror pool status' 2024-05-17T17:48:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T17:48:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool status --verbose 2024-05-17T17:53:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T17:53:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T17:53:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T17:53:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls 2024-05-17T17:58:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror_parent ns1 rbd_mirroring omap vals 2024-05-17T17:58:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror_parent ns1 rbd_mirroring omap vals' 2024-05-17T17:58:27.403 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2024-05-17T18:03:27.431 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-05-17T18:03:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror_parent ns1 rbd_mirror_leader omap vals 2024-05-17T18:03:27.432 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror_parent ns1 rbd_mirror_leader omap vals' 2024-05-17T18:03:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns1 listomapvals rbd_mirror_leader 2024-05-17T18:08:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-05-17T18:08:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T18:08:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T18:08:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2024-05-17T18:08:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror_parent ns2 images 2024-05-17T18:08:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror_parent ns2 images' 2024-05-17T18:08:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls -l 2024-05-17T18:13:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T18:13:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: listing images failed: (110) Connection timed out 2024-05-17T18:13:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T18:13:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T18:13:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror_parentns2 mirror pool info 2024-05-17T18:13:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror_parentns2 mirror pool info' 2024-05-17T18:13:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool info 2024-05-17T18:18:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T18:18:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T18:18:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T18:18:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror_parentns2 mirror pool status 2024-05-17T18:18:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror_parentns2 mirror pool status' 2024-05-17T18:18:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T18:18:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool status --verbose 2024-05-17T18:23:27.558 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:rbd: couldn't connect to the cluster! 2024-05-17T18:23:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T18:23:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T18:23:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls 2024-05-17T18:28:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror_parent ns2 rbd_mirroring omap vals 2024-05-17T18:28:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror_parent ns2 rbd_mirroring omap vals' 2024-05-17T18:28:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2024-05-17T18:33:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-05-17T18:33:27.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 mirror_parent ns2 rbd_mirror_leader omap vals 2024-05-17T18:33:27.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 mirror_parent ns2 rbd_mirror_leader omap vals' 2024-05-17T18:33:27.624 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns2 listomapvals rbd_mirror_leader 2024-05-17T18:38:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:failed to fetch mon config (--no-mon-config to skip) 2024-05-17T18:38:27.653 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout: 2024-05-17T18:38:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 2024-05-17T18:38:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local ret 2024-05-17T18:38:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-05-17T18:38:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ seq 0 3 2024-05-17T18:38:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T18:38:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ daemon_pid_file cluster1:0 2024-05-17T18:38:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:0 2024-05-17T18:38:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T18:38:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-05-17T18:38:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:0 2024-05-17T18:38:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T18:38:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T18:38:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T18:38:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T18:38:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster1:0 ']' 2024-05-17T18:38:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T18:38:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T18:38:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T18:38:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T18:38:27.657 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-05-17T18:38:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-05-17T18:38:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-05-17T18:38:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid ']' 2024-05-17T18:38:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid not exist) 2024-05-17T18:38:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid not exist)' 2024-05-17T18:38:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ continue 2024-05-17T18:38:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T18:38:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ daemon_pid_file cluster1:1 2024-05-17T18:38:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:1 2024-05-17T18:38:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T18:38:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-05-17T18:38:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:1 2024-05-17T18:38:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T18:38:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T18:38:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T18:38:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T18:38:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster1:1 ']' 2024-05-17T18:38:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T18:38:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T18:38:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T18:38:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T18:38:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-05-17T18:38:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-05-17T18:38:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-05-17T18:38:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid ']' 2024-05-17T18:38:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid not exist)' 2024-05-17T18:38:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid not exist) 2024-05-17T18:38:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ continue 2024-05-17T18:38:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T18:38:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ daemon_pid_file cluster1:2 2024-05-17T18:38:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:2 2024-05-17T18:38:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T18:38:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-05-17T18:38:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:2 2024-05-17T18:38:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T18:38:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T18:38:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T18:38:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T18:38:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster1:2 ']' 2024-05-17T18:38:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T18:38:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T18:38:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T18:38:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T18:38:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2024-05-17T18:38:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2024-05-17T18:38:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2024-05-17T18:38:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid ']' 2024-05-17T18:38:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid not exist)' 2024-05-17T18:38:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid not exist) 2024-05-17T18:38:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ continue 2024-05-17T18:38:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T18:38:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ daemon_pid_file cluster1:3 2024-05-17T18:38:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster1:3 2024-05-17T18:38:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T18:38:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-05-17T18:38:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster1:3 2024-05-17T18:38:27.723 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T18:38:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T18:38:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster1 2024-05-17T18:38:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T18:38:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster1:3 ']' 2024-05-17T18:38:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster1 2024-05-17T18:38:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster1 2024-05-17T18:38:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T18:38:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T18:38:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2024-05-17T18:38:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2024-05-17T18:38:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2024-05-17T18:38:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid ']' 2024-05-17T18:38:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid not exist)' 2024-05-17T18:38:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid not exist) 2024-05-17T18:38:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ continue 2024-05-17T18:38:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-05-17T18:38:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ seq 0 3 2024-05-17T18:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T18:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ daemon_pid_file cluster2:0 2024-05-17T18:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster2:0 2024-05-17T18:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T18:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-05-17T18:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster2:0 2024-05-17T18:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T18:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T18:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster2 2024-05-17T18:38:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=0 2024-05-17T18:38:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 0 = cluster2:0 ']' 2024-05-17T18:38:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster2 2024-05-17T18:38:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster2 2024-05-17T18:38:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=0 2024-05-17T18:38:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=0 2024-05-17T18:38:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-05-17T18:38:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-05-17T18:38:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-05-17T18:38:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid ']' 2024-05-17T18:38:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid not exist) 2024-05-17T18:38:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid not exist)' 2024-05-17T18:38:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ continue 2024-05-17T18:38:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T18:38:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ daemon_pid_file cluster2:1 2024-05-17T18:38:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster2:1 2024-05-17T18:38:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T18:38:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-05-17T18:38:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster2:1 2024-05-17T18:38:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T18:38:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T18:38:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster2 2024-05-17T18:38:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=1 2024-05-17T18:38:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 1 = cluster2:1 ']' 2024-05-17T18:38:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster2 2024-05-17T18:38:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster2 2024-05-17T18:38:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=1 2024-05-17T18:38:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=1 2024-05-17T18:38:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-05-17T18:38:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-05-17T18:38:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-05-17T18:38:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid ']' 2024-05-17T18:38:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid not exist)' 2024-05-17T18:38:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid not exist) 2024-05-17T18:38:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ continue 2024-05-17T18:38:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T18:38:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ daemon_pid_file cluster2:2 2024-05-17T18:38:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster2:2 2024-05-17T18:38:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T18:38:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster2:2 cluster instance 2024-05-17T18:38:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster2:2 2024-05-17T18:38:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T18:38:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T18:38:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster2 2024-05-17T18:38:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=2 2024-05-17T18:38:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 2 = cluster2:2 ']' 2024-05-17T18:38:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster2 2024-05-17T18:38:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster2 2024-05-17T18:38:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=2 2024-05-17T18:38:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=2 2024-05-17T18:38:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file' 2024-05-17T18:38:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2024-05-17T18:38:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2024-05-17T18:38:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid ']' 2024-05-17T18:38:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid not exist)' 2024-05-17T18:38:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid not exist) 2024-05-17T18:38:27.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ continue 2024-05-17T18:38:27.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T18:38:27.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ daemon_pid_file cluster2:3 2024-05-17T18:38:27.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster=cluster2:3 2024-05-17T18:38:27.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance 2024-05-17T18:38:27.807 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ set_cluster_instance cluster2:3 cluster instance 2024-05-17T18:38:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local val=cluster2:3 2024-05-17T18:38:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local cluster_var_name=cluster 2024-05-17T18:38:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ local instance_var_name=instance 2024-05-17T18:38:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ cluster=cluster2 2024-05-17T18:38:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ instance=3 2024-05-17T18:38:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ '[' 3 = cluster2:3 ']' 2024-05-17T18:38:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval cluster=cluster2 2024-05-17T18:38:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ cluster=cluster2 2024-05-17T18:38:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ eval instance=3 2024-05-17T18:38:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ instance=3 2024-05-17T18:38:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file' 2024-05-17T18:38:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2024-05-17T18:38:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2024-05-17T18:38:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid ']' 2024-05-17T18:38:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid not exist)' 2024-05-17T18:38:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid not exist) 2024-05-17T18:38:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ continue 2024-05-17T18:38:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 2024-05-17T18:38:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -z '' ']' 2024-05-17T18:38:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster instance 2024-05-17T18:38:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T18:38:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-05-17T18:43:27.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:2024-05-17T18:43:27.910+0000 7fef216e6640 0 monclient(hunting): authenticate timed out after 300 2024-05-17T18:43:27.904 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-05-17T18:43:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T18:43:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-05-17T18:48:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:2024-05-17T18:48:28.000+0000 7f262d6c3640 0 monclient(hunting): authenticate timed out after 300 2024-05-17T18:48:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-05-17T18:48:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T18:48:27.994 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-05-17T18:53:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:2024-05-17T18:53:28.090+0000 7fe7b3a95640 0 monclient(hunting): authenticate timed out after 300 2024-05-17T18:53:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-05-17T18:53:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ CEPH_ARGS= 2024-05-17T18:53:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-05-17T18:58:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:2024-05-17T18:58:28.180+0000 7fd6b7de8640 0 monclient(hunting): authenticate timed out after 300 2024-05-17T18:58:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-05-17T18:58:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-05-17T18:58:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stop_mirrors cluster1 2024-05-17T18:58:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1 2024-05-17T18:58:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local sig= 2024-05-17T18:58:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ seq 0 3 2024-05-17T18:58:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T18:58:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stop_mirror cluster1:0 '' 2024-05-17T18:58:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1:0 2024-05-17T18:58:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local sig= 2024-05-17T18:58:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n 1 2024-05-17T18:58:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 2024-05-17T18:58:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T18:58:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stop_mirror cluster1:1 '' 2024-05-17T18:58:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1:1 2024-05-17T18:58:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local sig= 2024-05-17T18:58:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n 1 2024-05-17T18:58:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 2024-05-17T18:58:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T18:58:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stop_mirror cluster1:2 '' 2024-05-17T18:58:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1:2 2024-05-17T18:58:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local sig= 2024-05-17T18:58:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n 1 2024-05-17T18:58:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 2024-05-17T18:58:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T18:58:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stop_mirror cluster1:3 '' 2024-05-17T18:58:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster1:3 2024-05-17T18:58:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local sig= 2024-05-17T18:58:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n 1 2024-05-17T18:58:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 2024-05-17T18:58:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-05-17T18:58:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stop_mirrors cluster2 2024-05-17T18:58:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2 2024-05-17T18:58:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local sig= 2024-05-17T18:58:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:++ seq 0 3 2024-05-17T18:58:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T18:58:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stop_mirror cluster2:0 '' 2024-05-17T18:58:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2:0 2024-05-17T18:58:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local sig= 2024-05-17T18:58:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n 1 2024-05-17T18:58:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 2024-05-17T18:58:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T18:58:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stop_mirror cluster2:1 '' 2024-05-17T18:58:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2:1 2024-05-17T18:58:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local sig= 2024-05-17T18:58:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n 1 2024-05-17T18:58:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 2024-05-17T18:58:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T18:58:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stop_mirror cluster2:2 '' 2024-05-17T18:58:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2:2 2024-05-17T18:58:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local sig= 2024-05-17T18:58:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n 1 2024-05-17T18:58:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 2024-05-17T18:58:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-05-17T18:58:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ stop_mirror cluster2:3 '' 2024-05-17T18:58:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local cluster=cluster2:3 2024-05-17T18:58:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ local sig= 2024-05-17T18:58:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test -n 1 2024-05-17T18:58:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ return 2024-05-17T18:58:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' -z 1 ']' 2024-05-17T18:58:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ test '' = /tmp/tmp.0i1dljubcH 2024-05-17T18:58:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ rm -Rf /tmp/tmp.0i1dljubcH 2024-05-17T18:58:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ '[' 110 -eq 0 ']' 2024-05-17T18:58:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ echo FAIL 2024-05-17T18:58:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stdout:FAIL 2024-05-17T18:58:28.185 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-05-17T18:58:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi195.stderr:+ exit 110 2024-05-17T18:58:28.186 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-05-17T18:58:28.186 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-05-17T18:58:28.508 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi195 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee59d86aad04089481e55c1a15aa5871339935b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2024-05-17T18:58:30.170 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e7d6cbdcf9eb45e18430628889b696e6 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi195 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee59d86aad04089481e55c1a15aa5871339935b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2024-05-17T18:58:30.172 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-05-17T18:58:30.206 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-05-17T18:58:30.207 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror-thrash Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2024-05-17T18:58:30.211 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-05-17T18:58:30.223 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-05-17T18:58:30.223 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror-thrash Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror_thrash.py", line 211, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2024-05-17T18:58:30.229 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-05-17T18:58:30.242 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-05-17T18:58:30.242 INFO:teuthology.orchestra.run:waiting for 300 2024-05-17T18:58:30.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi195.stderr:2024-05-17T18:58:30.253+0000 7fa1725c4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 51715) UID: 1000 2024-05-17T19:03:29.308 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-05-17T19:03:29.315 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-05-17T19:03:29.326 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-05-17T19:03:29.326 INFO:teuthology.orchestra.run:waiting for 300 2024-05-17T19:03:29.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi195.stderr:2024-05-17T19:03:29.338+0000 7f272455d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 50187) UID: 1000 2024-05-17T19:08:28.404 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-05-17T19:08:28.410 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-05-17T19:08:28.421 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-05-17T19:08:28.421 INFO:teuthology.orchestra.run:waiting for 300 2024-05-17T19:08:28.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi195.stderr:2024-05-17T19:08:28.433+0000 7f4c5ae78640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 58679) UID: 1000 2024-05-17T19:13:27.496 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-05-17T19:13:27.502 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-05-17T19:13:27.513 ERROR:tasks.rbd_mirror.cluster2.client.mirror.0:tried to stop a non-running daemon 2024-05-17T19:13:27.516 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-05-17T19:13:27.528 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-05-17T19:13:27.528 INFO:teuthology.orchestra.run:waiting for 300 2024-05-17T19:13:27.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi195.stderr:2024-05-17T19:13:27.540+0000 7fd0d98b7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 57118) UID: 1000 2024-05-17T19:18:26.602 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-05-17T19:18:26.609 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-05-17T19:18:26.621 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-05-17T19:18:26.621 INFO:teuthology.orchestra.run:waiting for 300 2024-05-17T19:18:26.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi195.stderr:2024-05-17T19:18:26.629+0000 7f311079d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 47162) UID: 1000 2024-05-17T19:23:25.699 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-05-17T19:23:25.708 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-05-17T19:23:25.724 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-05-17T19:23:25.724 INFO:teuthology.orchestra.run:waiting for 300 2024-05-17T19:23:25.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi195.stderr:2024-05-17T19:23:25.729+0000 7f8ccd849640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 54464) UID: 1000 2024-05-17T19:28:24.796 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-05-17T19:28:24.804 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-05-17T19:28:24.819 ERROR:tasks.rbd_mirror.cluster1.client.mirror.0:tried to stop a non-running daemon 2024-05-17T19:28:24.822 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-17T19:28:24.888 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-05-17T19:28:24.888 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-05-17T19:30:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-05-17T19:30:24.931 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2024-05-17T19:30:24.947 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-17T19:30:24.947 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-17T19:30:24.947 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-05-17T19:30:24.948 INFO:teuthology.orchestra.run:waiting for 300 2024-05-17T19:30:24.951 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-05-17T19:30:24.951 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0' 2024-05-17T19:30:24.963 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-05-17T19:30:24.963 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-05-17T19:30:24.963 INFO:teuthology.orchestra.run:waiting for 300 2024-05-17T19:30:24.964 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-05-17T19:30:24.964 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-05-17T19:30:24.964 INFO:teuthology.orchestra.run:waiting for 300 2024-05-17T19:30:24.964 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-05-17T19:30:24.965 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-17T19:30:24.965 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-05-17T19:30:24.965 INFO:teuthology.orchestra.run:waiting for 300 2024-05-17T19:30:24.965 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-05-17T19:30:24.965 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-17T19:30:24.965 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-05-17T19:30:24.965 INFO:teuthology.orchestra.run:waiting for 300 2024-05-17T19:30:24.966 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-05-17T19:30:24.966 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-17T19:30:24.966 DEBUG:teuthology.orchestra.run.smithi195:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-05-17T19:30:25.020 INFO:teuthology.orchestra.run.smithi195.stdout:2024-05-17T15:13:06.552274+0000 mon.a (mon.0) 799 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-05-17T19:30:25.021 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-05-17T19:30:25.021 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi195.front.sepia.ceph.com 2024-05-17T19:30:25.022 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-05-17T19:30:25.279 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi195.front.sepia.ceph.com 2024-05-17T19:30:25.279 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-05-17T19:30:25.475 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi195.front.sepia.ceph.com 2024-05-17T19:30:25.476 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-05-17T19:30:25.638 INFO:tasks.ceph:Archiving mon data... 2024-05-17T19:30:25.640 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219/data/cluster2.mon.a.tgz 2024-05-17T19:30:25.640 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2024-05-17T19:30:25.656 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.2k6KySuIl1 2024-05-17T19:30:25.656 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.2k6KySuIl1 2024-05-17T19:30:25.813 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.2k6KySuIl1 2024-05-17T19:30:25.909 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.2k6KySuIl1 is 433KB 2024-05-17T19:30:26.007 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.2k6KySuIl1 2024-05-17T19:30:26.023 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-17T19:30:26.023 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-05-17T19:30:26.026 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-05-17T19:30:26.087 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2024-05-17T19:30:26.098 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-17T19:30:26.111 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-05-17T19:30:26.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-05-17T19:32:26.149 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-05-17T19:32:26.151 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json' 2024-05-17T19:32:26.167 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-17T19:32:26.167 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-17T19:32:26.168 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-05-17T19:32:26.168 INFO:teuthology.orchestra.run:waiting for 300 2024-05-17T19:32:26.172 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-05-17T19:32:26.172 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-05-17T19:32:26.172 INFO:teuthology.orchestra.run:waiting for 300 2024-05-17T19:32:26.173 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-05-17T19:32:26.173 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-05-17T19:32:26.173 INFO:teuthology.orchestra.run:waiting for 300 2024-05-17T19:32:26.173 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-05-17T19:32:26.173 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-17T19:32:26.174 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-05-17T19:32:26.174 INFO:teuthology.orchestra.run:waiting for 300 2024-05-17T19:32:26.174 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-05-17T19:32:26.174 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-17T19:32:26.174 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-05-17T19:32:26.175 INFO:teuthology.orchestra.run:waiting for 300 2024-05-17T19:32:26.175 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-05-17T19:32:26.175 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-17T19:32:26.176 DEBUG:teuthology.orchestra.run.smithi134:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-05-17T19:32:26.345 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi134.front.sepia.ceph.com 2024-05-17T19:32:26.345 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-05-17T19:32:26.419 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi134.front.sepia.ceph.com 2024-05-17T19:32:26.419 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-05-17T19:32:26.494 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi134.front.sepia.ceph.com 2024-05-17T19:32:26.494 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-05-17T19:32:26.558 INFO:tasks.ceph:Archiving mon data... 2024-05-17T19:32:26.558 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219/data/cluster1.mon.a.tgz 2024-05-17T19:32:26.558 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-05-17T19:32:26.574 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.4Vj9r1PGEX 2024-05-17T19:32:26.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.4Vj9r1PGEX 2024-05-17T19:32:26.724 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.4Vj9r1PGEX 2024-05-17T19:32:26.807 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.4Vj9r1PGEX is 297KB 2024-05-17T19:32:26.873 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.4Vj9r1PGEX 2024-05-17T19:32:26.888 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-17T19:32:26.888 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-05-17T19:32:26.930 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-05-17T19:32:27.008 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-17T19:32:27.026 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-17T19:32:27.026 INFO:tasks.ceph:Archiving crash dumps... 2024-05-17T19:32:27.028 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219/remote/smithi134/crash 2024-05-17T19:32:27.030 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-17T19:32:27.072 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219/remote/smithi195/crash 2024-05-17T19:32:27.073 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-17T19:32:27.146 INFO:tasks.ceph:Compressing logs... 2024-05-17T19:32:27.146 DEBUG:teuthology.orchestra.run.smithi134:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-17T19:32:27.149 DEBUG:teuthology.orchestra.run.smithi195:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-17T19:32:27.170 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37648.log 2024-05-17T19:32:27.171 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32581.log 2024-05-17T19:32:27.171 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.37648.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-05-17T19:32:27.171 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37648.log.gz 2024-05-17T19:32:27.171 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.32581.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/cluster1-client.admin.32581.log.gz 2024-05-17T19:32:27.171 INFO:teuthology.orchestra.run.smithi134.stderr: /var/log/ceph/ceph.tmp-client.admin.27904.log 2024-05-17T19:32:27.181 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37606.log 2024-05-17T19:32:27.181 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph.tmp-client.admin.27904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37498.log 2024-05-17T19:32:27.181 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27904.log.gz 2024-05-17T19:32:27.182 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.37606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37226.log 2024-05-17T19:32:27.182 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37606.log.gz 2024-05-17T19:32:27.182 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.37498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37144.log 2024-05-17T19:32:27.182 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37498.log.gz 2024-05-17T19:32:27.182 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.37226.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37226.log.gz 2024-05-17T19:32:27.182 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37018.log 2024-05-17T19:32:27.182 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.37144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32720.log 2024-05-17T19:32:27.182 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37144.log.gz 2024-05-17T19:32:27.182 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.37018.log: gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-05-17T19:32:27.183 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37018.log.gz 2024-05-17T19:32:27.183 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.32720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32541.log 2024-05-17T19:32:27.183 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32720.log.gz 2024-05-17T19:32:27.183 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37438.log 2024-05-17T19:32:27.183 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.32541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37312.log 0.0% 2024-05-17T19:32:27.183 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.32541.log.gz 2024-05-17T19:32:27.183 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.37438.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32621.log 2024-05-17T19:32:27.183 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37438.log.gz 2024-05-17T19:32:27.184 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.37312.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37312.log.gz 2024-05-17T19:32:27.184 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-05-17T19:32:27.184 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36976.log 2024-05-17T19:32:27.184 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.32621.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32621.log.gz 2024-05-17T19:32:27.184 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37060.log 2024-05-17T19:32:27.184 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.36976.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-05-17T19:32:27.184 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36976.log.gz 2024-05-17T19:32:27.184 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.37060.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37060.log.gz 2024-05-17T19:32:27.184 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36836.log 2024-05-17T19:32:27.185 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37102.log 2024-05-17T19:32:27.185 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1.audit.log: /var/log/ceph/cluster1-client.admin.36836.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36836.log.gz 2024-05-17T19:32:27.185 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37351.log 2024-05-17T19:32:27.185 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-05-17T19:32:27.185 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.37102.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37102.log.gz 2024-05-17T19:32:27.185 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.37351.log: 89.0% -- replaced with /var/log/ceph/cluster1.log.gz 2024-05-17T19:32:27.185 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-05-17T19:32:27.185 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37351.log.gz 2024-05-17T19:32:27.185 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37564.log 2024-05-17T19:32:27.185 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32793.log 2024-05-17T19:32:27.186 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.37564.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37564.log.gz 2024-05-17T19:32:27.186 INFO:teuthology.orchestra.run.smithi134.stderr: -- /var/log/ceph/cluster1-client.admin.37458.log 2024-05-17T19:32:27.186 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.32793.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32793.log.gz 2024-05-17T19:32:27.186 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37224.log 2024-05-17T19:32:27.186 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.37458.log: 92.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36920.log 2024-05-17T19:32:27.186 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-05-17T19:32:27.186 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.37458.log.gz 2024-05-17T19:32:27.186 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.37224.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32383.log 2024-05-17T19:32:27.187 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37224.log.gz 2024-05-17T19:32:27.187 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.36920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37225.log 2024-05-17T19:32:27.187 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36920.log.gz 2024-05-17T19:32:27.187 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.32383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-05-17T19:32:27.187 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32383.log.gz 2024-05-17T19:32:27.187 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.37225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39572.log 2024-05-17T19:32:27.187 INFO:teuthology.orchestra.run.smithi134.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37225.log.gz 2024-05-17T19:32:27.187 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-05-17T19:32:27.187 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-client.admin.39572.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39572.log.gz 2024-05-17T19:32:27.188 INFO:teuthology.orchestra.run.smithi134.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.37352.log 2024-05-17T19:32:27.188 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cluster1-osd.2.log: /var/log/ceph/cluster1-client.admin.37352.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37352.log.gz 2024-05-17T19:32:27.205 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42399.log 2024-05-17T19:32:27.205 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63332.log 2024-05-17T19:32:27.205 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.42399.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45975.log 2024-05-17T19:32:27.207 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63332.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42399.log.gzgzip 2024-05-17T19:32:27.207 INFO:teuthology.orchestra.run.smithi195.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43960.log 2024-05-17T19:32:27.207 INFO:teuthology.orchestra.run.smithi195.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63332.log.gz 2024-05-17T19:32:27.207 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58437.log 2024-05-17T19:32:27.207 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.43960.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56068.log 2024-05-17T19:32:27.207 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53775.log 2024-05-17T19:32:27.207 INFO:teuthology.orchestra.run.smithi195.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58437.log.gz 2024-05-17T19:32:27.207 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.56068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47120.log 2024-05-17T19:32:27.212 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49538.log 2024-05-17T19:32:27.213 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53775.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56068.log.gz 2024-05-17T19:32:27.213 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47120.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62868.log 2024-05-17T19:32:27.213 INFO:teuthology.orchestra.run.smithi195.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53775.log.gz 2024-05-17T19:32:27.213 INFO:teuthology.orchestra.run.smithi195.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47120.log.gz 2024-05-17T19:32:27.213 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63015.log 2024-05-17T19:32:27.213 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49538.log: /var/log/ceph/cluster1-client.mirror.62868.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62868.log.gz 2024-05-17T19:32:27.213 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61653.log 2024-05-17T19:32:27.213 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61833.log 2024-05-17T19:32:27.214 INFO:teuthology.orchestra.run.smithi195.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49538.log.gz 2024-05-17T19:32:27.214 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63015.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63015.log.gz 2024-05-17T19:32:27.214 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61653.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49659.log 2024-05-17T19:32:27.214 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61833.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61653.log.gz 2024-05-17T19:32:27.214 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45179.log 2024-05-17T19:32:27.214 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43824.log 2024-05-17T19:32:27.214 INFO:teuthology.orchestra.run.smithi195.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61833.log.gz 2024-05-17T19:32:27.214 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49659.log: /var/log/ceph/cluster1-client.mirror.45179.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62705.log 2024-05-17T19:32:27.215 INFO:teuthology.orchestra.run.smithi195.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43960.log.gz 2024-05-17T19:32:27.215 INFO:teuthology.orchestra.run.smithi195.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45179.log.gz 2024-05-17T19:32:27.215 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.43824.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37741.log 2024-05-17T19:32:27.215 INFO:teuthology.orchestra.run.smithi195.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45975.log.gz 2024-05-17T19:32:27.215 INFO:teuthology.orchestra.run.smithi195.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43824.log.gz 2024-05-17T19:32:27.215 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48475.log 2024-05-17T19:32:27.215 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62705.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62705.log.gz 2024-05-17T19:32:27.215 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51880.log 2024-05-17T19:32:27.215 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.37741.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37741.log.gz 2024-05-17T19:32:27.216 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48475.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49347.log 2024-05-17T19:32:27.216 INFO:teuthology.orchestra.run.smithi195.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48475.log.gz 2024-05-17T19:32:27.216 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61993.log 2024-05-17T19:32:27.216 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49347.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56712.log 2024-05-17T19:32:27.216 INFO:teuthology.orchestra.run.smithi195.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49347.log.gz 2024-05-17T19:32:27.216 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61993.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44537.log 2024-05-17T19:32:27.216 INFO:teuthology.orchestra.run.smithi195.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61993.log.gz 2024-05-17T19:32:27.216 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60313.log 2024-05-17T19:32:27.216 INFO:teuthology.orchestra.run.smithi195.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56712.log.gz 2024-05-17T19:32:27.217 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.44537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40572.log 2024-05-17T19:32:27.228 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40129.log 2024-05-17T19:32:27.228 INFO:teuthology.orchestra.run.smithi195.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60313.log.gz 2024-05-17T19:32:27.228 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.40572.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50080.log 2024-05-17T19:32:27.228 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.40129.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44285.log 2024-05-17T19:32:27.228 INFO:teuthology.orchestra.run.smithi195.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44537.log.gz 2024-05-17T19:32:27.228 INFO:teuthology.orchestra.run.smithi195.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.40129.log.gz 2024-05-17T19:32:27.228 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50080.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46247.log 2024-05-17T19:32:27.229 INFO:teuthology.orchestra.run.smithi195.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40572.log.gz 2024-05-17T19:32:27.229 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61393.log 2024-05-17T19:32:27.229 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.44285.log: /var/log/ceph/cluster2-client.mirror.46247.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50774.log 2024-05-17T19:32:27.229 INFO:teuthology.orchestra.run.smithi195.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44285.log.gz 2024-05-17T19:32:27.229 INFO:teuthology.orchestra.run.smithi195.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50080.log.gz 2024-05-17T19:32:27.229 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61393.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48270.log 2024-05-17T19:32:27.229 INFO:teuthology.orchestra.run.smithi195.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46247.log.gz 2024-05-17T19:32:27.229 INFO:teuthology.orchestra.run.smithi195.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61393.log.gz 2024-05-17T19:32:27.229 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50774.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62922.log 2024-05-17T19:32:27.230 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58209.log 2024-05-17T19:32:27.230 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48270.log: /var/log/ceph/cluster2-client.admin.62922.log: gzip 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.62922.log.gz -5 2024-05-17T19:32:27.230 INFO:teuthology.orchestra.run.smithi195.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.55590.log 2024-05-17T19:32:27.230 INFO:teuthology.orchestra.run.smithi195.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50774.log.gz 2024-05-17T19:32:27.230 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.58209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44237.log 2024-05-17T19:32:27.230 INFO:teuthology.orchestra.run.smithi195.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48270.log.gz 2024-05-17T19:32:27.230 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.55590.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55244.log 2024-05-17T19:32:27.230 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.44237.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61453.log 2024-05-17T19:32:27.231 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55244.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55590.log.gz 2024-05-17T19:32:27.231 INFO:teuthology.orchestra.run.smithi195.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44237.log.gz 2024-05-17T19:32:27.231 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58945.log 2024-05-17T19:32:27.231 INFO:teuthology.orchestra.run.smithi195.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55244.log.gz 2024-05-17T19:32:27.231 INFO:teuthology.orchestra.run.smithi195.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58209.log.gz 2024-05-17T19:32:27.231 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57001.log 2024-05-17T19:32:27.231 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61453.log: /var/log/ceph/cluster1-client.mirror.58945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45737.log 2024-05-17T19:32:27.231 INFO:teuthology.orchestra.run.smithi195.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61453.log.gz 2024-05-17T19:32:27.232 INFO:teuthology.orchestra.run.smithi195.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58945.log.gz 2024-05-17T19:32:27.232 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57001.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37361.log 2024-05-17T19:32:27.232 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45737.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57001.log.gz 2024-05-17T19:32:27.232 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63089.log 2024-05-17T19:32:27.232 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.37361.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37361.log.gz 2024-05-17T19:32:27.232 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47898.log 2024-05-17T19:32:27.232 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46347.log 2024-05-17T19:32:27.232 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63089.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63089.log.gz 2024-05-17T19:32:27.232 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62313.log 2024-05-17T19:32:27.233 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46347.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51838.log 2024-05-17T19:32:27.233 INFO:teuthology.orchestra.run.smithi195.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47898.log.gz 2024-05-17T19:32:27.233 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63191.log 2024-05-17T19:32:27.233 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62313.log.gz 2024-05-17T19:32:27.233 INFO:teuthology.orchestra.run.smithi195.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46347.log.gz 2024-05-17T19:32:27.233 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51838.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63179.log 2024-05-17T19:32:27.233 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63191.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58068.log 2024-05-17T19:32:27.233 INFO:teuthology.orchestra.run.smithi195.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63191.log.gz 2024-05-17T19:32:27.234 INFO:teuthology.orchestra.run.smithi195.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51838.log.gz 2024-05-17T19:32:27.234 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.63179.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55485.log 2024-05-17T19:32:27.234 INFO:teuthology.orchestra.run.smithi195.stderr: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.63179.log.gz 2024-05-17T19:32:27.234 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.58068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62253.log 2024-05-17T19:32:27.234 INFO:teuthology.orchestra.run.smithi195.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58068.log.gz 2024-05-17T19:32:27.234 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55485.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43570.log 2024-05-17T19:32:27.234 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37360.log 2024-05-17T19:32:27.234 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62253.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55485.log.gz 2024-05-17T19:32:27.235 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.43570.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42941.log 2024-05-17T19:32:27.235 INFO:teuthology.orchestra.run.smithi195.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62253.log.gz 2024-05-17T19:32:27.235 INFO:teuthology.orchestra.run.smithi195.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45737.log.gz 2024-05-17T19:32:27.235 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.37360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63036.log 2024-05-17T19:32:27.235 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37360.log.gz 2024-05-17T19:32:27.235 INFO:teuthology.orchestra.run.smithi195.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43570.log.gz 2024-05-17T19:32:27.235 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50397.log 2024-05-17T19:32:27.235 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.42941.log: /var/log/ceph/cluster2-client.mirror.63036.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52924.log 2024-05-17T19:32:27.235 INFO:teuthology.orchestra.run.smithi195.stderr: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63036.log.gz 2024-05-17T19:32:27.235 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50397.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61773.log 2024-05-17T19:32:27.236 INFO:teuthology.orchestra.run.smithi195.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50397.log.gz 2024-05-17T19:32:27.236 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.52924.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46675.log 2024-05-17T19:32:27.236 INFO:teuthology.orchestra.run.smithi195.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52924.log.gz 2024-05-17T19:32:27.236 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61773.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36911.log 2024-05-17T19:32:27.236 INFO:teuthology.orchestra.run.smithi195.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61773.log.gz 2024-05-17T19:32:27.236 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46675.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51923.log 2024-05-17T19:32:27.236 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.36911.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36911.log.gz 2024-05-17T19:32:27.236 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50456.log 2024-05-17T19:32:27.237 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47475.log 2024-05-17T19:32:27.237 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51923.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46675.log.gz 2024-05-17T19:32:27.237 INFO:teuthology.orchestra.run.smithi195.stderr: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51923.log.gz 2024-05-17T19:32:27.237 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50456.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45264.log 2024-05-17T19:32:27.237 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50994.log 2024-05-17T19:32:27.237 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47475.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50456.log.gz 2024-05-17T19:32:27.237 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45264.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45502.log 2024-05-17T19:32:27.237 INFO:teuthology.orchestra.run.smithi195.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47475.log.gz 2024-05-17T19:32:27.260 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62133.log 2024-05-17T19:32:27.261 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50994.log: /var/log/ceph/cluster2-client.mirror.45502.log: 80.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56535.log 2024-05-17T19:32:27.261 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.50994.log.gz 2024-05-17T19:32:27.261 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62133.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.57139.log 2024-05-17T19:32:27.261 INFO:teuthology.orchestra.run.smithi195.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48131.log 2024-05-17T19:32:27.261 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.62133.log.gz 2024-05-17T19:32:27.261 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56535.log: /var/log/ceph/cluster1-client.mirror.3.57139.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56535.log.gz 2024-05-17T19:32:27.261 INFO:teuthology.orchestra.run.smithi195.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42941.log.gz 2024-05-17T19:32:27.261 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44781.log 2024-05-17T19:32:27.262 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.38195.log 2024-05-17T19:32:27.262 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.44781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45155.log 2024-05-17T19:32:27.262 INFO:teuthology.orchestra.run.smithi195.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44781.log.gz 2024-05-17T19:32:27.262 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.1.38195.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45264.log.gz 2024-05-17T19:32:27.262 INFO:teuthology.orchestra.run.smithi195.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45502.log.gz 2024-05-17T19:32:27.262 INFO:teuthology.orchestra.run.smithi195.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.38195.log.gz 2024-05-17T19:32:27.262 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48003.log 2024-05-17T19:32:27.262 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45155.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48962.log 2024-05-17T19:32:27.262 INFO:teuthology.orchestra.run.smithi195.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45155.log.gz 2024-05-17T19:32:27.263 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57988.log 2024-05-17T19:32:27.263 INFO:teuthology.orchestra.run.smithi195.stderr:gzip/var/log/ceph/cluster2-client.mirror.48962.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61213.log 2024-05-17T19:32:27.263 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57988.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48962.log.gz 2024-05-17T19:32:27.263 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60573.log 2024-05-17T19:32:27.263 INFO:teuthology.orchestra.run.smithi195.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48003.log.gz 2024-05-17T19:32:27.263 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61213.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61213.log.gz 2024-05-17T19:32:27.263 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57425.log 2024-05-17T19:32:27.263 INFO:teuthology.orchestra.run.smithi195.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57988.log.gz 2024-05-17T19:32:27.264 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60573.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60693.log 2024-05-17T19:32:27.264 INFO:teuthology.orchestra.run.smithi195.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60573.log.gz 2024-05-17T19:32:27.264 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57425.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55980.log 2024-05-17T19:32:27.264 INFO:teuthology.orchestra.run.smithi195.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57425.log.gz 2024-05-17T19:32:27.264 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46799.log 2024-05-17T19:32:27.264 INFO:teuthology.orchestra.run.smithi195.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60693.log.gz 2024-05-17T19:32:27.264 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.55980.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59873.log 2024-05-17T19:32:27.264 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46799.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33454.log 2024-05-17T19:32:27.264 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59873.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52169.log 2024-05-17T19:32:27.265 INFO:teuthology.orchestra.run.smithi195.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59873.log.gz 2024-05-17T19:32:27.265 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.33454.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55980.log.gz 2024-05-17T19:32:27.265 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33454.log.gz 2024-05-17T19:32:27.265 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49083.log 2024-05-17T19:32:27.265 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.52169.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61593.log 2024-05-17T19:32:27.265 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49083.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47268.log 2024-05-17T19:32:27.265 INFO:teuthology.orchestra.run.smithi195.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49083.log.gz 2024-05-17T19:32:27.265 INFO:teuthology.orchestra.run.smithi195.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52169.log.gz 2024-05-17T19:32:27.266 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61593.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53348.log 2024-05-17T19:32:27.266 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47268.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47268.log.gz 2024-05-17T19:32:27.266 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.53348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52659.log 2024-05-17T19:32:27.266 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- 89.6% /var/log/ceph/cluster1-client.mirror.62053.log 2024-05-17T19:32:27.266 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.53348.log.gz 2024-05-17T19:32:27.266 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43440.log 2024-05-17T19:32:27.266 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.52659.log: /var/log/ceph/cluster1-client.mirror.62053.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61593.log.gz 2024-05-17T19:32:27.266 INFO:teuthology.orchestra.run.smithi195.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62053.log.gz 2024-05-17T19:32:27.266 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52948.log 2024-05-17T19:32:27.267 INFO:teuthology.orchestra.run.smithi195.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52659.log.gz 2024-05-17T19:32:27.267 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.43440.log: gzip -5 --verbose -- 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43440.log.gz 2024-05-17T19:32:27.267 INFO:teuthology.orchestra.run.smithi195.stderr: /var/log/ceph/cluster1-client.mirror.40285.log 2024-05-17T19:32:27.267 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52948.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42963.log 2024-05-17T19:32:27.267 INFO:teuthology.orchestra.run.smithi195.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52948.log.gz 2024-05-17T19:32:27.267 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.40285.log: gzip -5 --verbose -- 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40285.log.gz 2024-05-17T19:32:27.267 INFO:teuthology.orchestra.run.smithi195.stderr: /var/log/ceph/cluster1-client.mirror.58788.log 2024-05-17T19:32:27.267 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.42963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60153.log 2024-05-17T19:32:27.268 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58788.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56000.log 2024-05-17T19:32:27.268 INFO:teuthology.orchestra.run.smithi195.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42963.log.gz 2024-05-17T19:32:27.268 INFO:teuthology.orchestra.run.smithi195.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58788.log.gz 2024-05-17T19:32:27.268 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60153.log: gzip -5 --verbose -- 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60153.log.gz 2024-05-17T19:32:27.268 INFO:teuthology.orchestra.run.smithi195.stderr: /var/log/ceph/cluster2-client.mirror.49364.log 2024-05-17T19:32:27.268 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56000.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45087.log 2024-05-17T19:32:27.268 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44689.log 2024-05-17T19:32:27.268 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45087.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49364.log.gz 2024-05-17T19:32:27.268 INFO:teuthology.orchestra.run.smithi195.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56000.log.gz 2024-05-17T19:32:27.268 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45717.log 2024-05-17T19:32:27.269 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.44689.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45087.log.gz 2024-05-17T19:32:27.269 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63220.log 2024-05-17T19:32:27.269 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55088.log 2024-05-17T19:32:27.269 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63220.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63220.log.gz 2024-05-17T19:32:27.269 INFO:teuthology.orchestra.run.smithi195.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44689.log.gz 2024-05-17T19:32:27.269 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61153.log 2024-05-17T19:32:27.269 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62407.log 2024-05-17T19:32:27.269 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61153.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61153.log.gz 2024-05-17T19:32:27.269 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59373.log 2024-05-17T19:32:27.270 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62407.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55088.log.gz 2024-05-17T19:32:27.270 INFO:teuthology.orchestra.run.smithi195.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62407.log.gz 2024-05-17T19:32:27.270 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48312.log 2024-05-17T19:32:27.270 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59373.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39154.log 2024-05-17T19:32:27.270 INFO:teuthology.orchestra.run.smithi195.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59373.log.gz 2024-05-17T19:32:27.270 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45995.log 2024-05-17T19:32:27.270 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.39154.log: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39154.log.gz 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45717.log.gz 2024-05-17T19:32:27.270 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-05-17T19:32:27.271 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47438.log 2024-05-17T19:32:27.271 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45995.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41630.log 2024-05-17T19:32:27.271 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47438.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47438.log.gz 2024-05-17T19:32:27.271 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51053.log 2024-05-17T19:32:27.272 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.41630.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45995.log.gz 2024-05-17T19:32:27.272 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.38159.log 2024-05-17T19:32:27.272 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51053.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41630.log.gz 2024-05-17T19:32:27.272 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42620.log 2024-05-17T19:32:27.277 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.3.38159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41566.log 2024-05-17T19:32:27.278 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.42620.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.42620.log.gz 2024-05-17T19:32:27.292 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49834.log 2024-05-17T19:32:27.292 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.41566.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41566.log.gz 2024-05-17T19:32:27.304 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.38248.log 2024-05-17T19:32:27.305 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49834.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49834.log.gz 2024-05-17T19:32:27.318 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28543.log 2024-05-17T19:32:27.336 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.3.38248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49179.log 2024-05-17T19:32:27.336 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph.tmp-client.admin.28543.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28543.log.gz 2024-05-17T19:32:27.344 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59190.log 2024-05-17T19:32:27.350 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49179.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.3.38248.log.gz 2024-05-17T19:32:27.358 INFO:teuthology.orchestra.run.smithi195.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49179.log.gz 2024-05-17T19:32:27.361 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44415.log 2024-05-17T19:32:27.361 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.59190.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.59190.log.gz 2024-05-17T19:32:27.363 INFO:teuthology.orchestra.run.smithi195.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.3.38159.log.gz 2024-05-17T19:32:27.364 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63312.log 2024-05-17T19:32:27.365 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.44415.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44415.log.gz 2024-05-17T19:32:27.366 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56314.log 2024-05-17T19:32:27.367 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.63312.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.63312.log.gz 2024-05-17T19:32:27.367 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41446.log 2024-05-17T19:32:27.368 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56314.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56314.log.gz 2024-05-17T19:32:27.368 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40080.log 2024-05-17T19:32:27.369 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.41446.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41446.log.gz 2024-05-17T19:32:27.370 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50932.log 2024-05-17T19:32:27.371 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.40080.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.40080.log.gz 2024-05-17T19:32:27.371 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60533.log 2024-05-17T19:32:27.372 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50932.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50932.log.gz 2024-05-17T19:32:27.373 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62816.log 2024-05-17T19:32:27.374 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60533.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60533.log.gz 2024-05-17T19:32:27.374 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40785.log 2024-05-17T19:32:27.376 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62816.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62816.log.gz 2024-05-17T19:32:27.376 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57807.log 2024-05-17T19:32:27.377 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.40785.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40785.log.gz 2024-05-17T19:32:27.377 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47775.log 2024-05-17T19:32:27.378 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.57807.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57807.log.gz 2024-05-17T19:32:27.379 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43368.log 2024-05-17T19:32:27.395 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47775.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52100.log 2024-05-17T19:32:27.396 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.43368.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43368.log.gz 2024-05-17T19:32:27.398 INFO:teuthology.orchestra.run.smithi134.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-05-17T19:32:27.412 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54223.log 2024-05-17T19:32:27.412 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52100.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52100.log.gz 2024-05-17T19:32:27.424 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62093.log 2024-05-17T19:32:27.425 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.54223.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54223.log.gz 2024-05-17T19:32:27.431 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36953.log 2024-05-17T19:32:27.431 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62093.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62093.log.gz 2024-05-17T19:32:27.441 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62722.log 2024-05-17T19:32:27.442 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.36953.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36953.log.gz 2024-05-17T19:32:27.448 INFO:teuthology.orchestra.run.smithi134.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-05-17T19:32:27.454 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46467.log 2024-05-17T19:32:27.455 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.62722.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.admin.62722.log.gz 2024-05-17T19:32:27.469 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52613.log 2024-05-17T19:32:27.470 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46467.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46467.log.gz 2024-05-17T19:32:27.484 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45608.log 2024-05-17T19:32:27.485 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.52613.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52613.log.gz 2024-05-17T19:32:27.499 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46568.log 2024-05-17T19:32:27.505 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45608.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45608.log.gz 2024-05-17T19:32:27.520 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45041.log 2024-05-17T19:32:27.521 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46568.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46568.log.gz 2024-05-17T19:32:27.533 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36825.log 2024-05-17T19:32:27.534 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45041.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45041.log.gz 2024-05-17T19:32:27.542 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61273.log 2024-05-17T19:32:27.551 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.36825.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36825.log.gz 2024-05-17T19:32:27.563 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53145.log 2024-05-17T19:32:27.564 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61273.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61273.log.gz 2024-05-17T19:32:27.584 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40809.log 2024-05-17T19:32:27.585 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.53145.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53145.log.gz 2024-05-17T19:32:27.593 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55526.log 2024-05-17T19:32:27.594 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.40809.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51053.log.gz 2024-05-17T19:32:27.595 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42717.log 2024-05-17T19:32:27.597 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55526.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55526.log.gz 2024-05-17T19:32:27.597 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41470.log 2024-05-17T19:32:27.599 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.42717.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42717.log.gz 2024-05-17T19:32:27.599 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51573.log 2024-05-17T19:32:27.600 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.41470.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.41470.log.gz 2024-05-17T19:32:27.600 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61793.log 2024-05-17T19:32:27.602 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51573.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51573.log.gz 2024-05-17T19:32:27.602 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58317.log 2024-05-17T19:32:27.603 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61793.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61793.log.gz 2024-05-17T19:32:27.604 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63128.log 2024-05-17T19:32:27.604 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.58317.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58317.log.gz 2024-05-17T19:32:27.605 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50678.log 2024-05-17T19:32:27.606 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63128.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63128.log.gz 2024-05-17T19:32:27.607 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40809.log.gz 2024-05-17T19:32:27.607 INFO:teuthology.orchestra.run.smithi195.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.0.41312.log 2024-05-17T19:32:27.607 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50678.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50678.log.gz 2024-05-17T19:32:27.608 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46633.log 2024-05-17T19:32:27.614 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.0.41312.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.41312.log.gz 2024-05-17T19:32:27.615 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47716.log 2024-05-17T19:32:27.616 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.46633.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.46633.log.gz 2024-05-17T19:32:27.616 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59693.log 2024-05-17T19:32:27.617 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47716.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47716.log.gz 2024-05-17T19:32:27.617 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50616.log 2024-05-17T19:32:27.618 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59693.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59693.log.gz 2024-05-17T19:32:27.618 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44976.log 2024-05-17T19:32:27.619 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50616.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50616.log.gz 2024-05-17T19:32:27.620 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57473.log 2024-05-17T19:32:27.626 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.44976.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44976.log.gz 2024-05-17T19:32:27.627 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43548.log 2024-05-17T19:32:27.627 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57473.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57473.log.gz 2024-05-17T19:32:27.629 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59733.log 2024-05-17T19:32:27.629 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.46799.log.gz 2024-05-17T19:32:27.629 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.43548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51944.log 2024-05-17T19:32:27.631 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59733.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59733.log.gz 2024-05-17T19:32:27.631 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48883.log 2024-05-17T19:32:27.631 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.51944.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51944.log.gz 2024-05-17T19:32:27.631 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56439.log 2024-05-17T19:32:27.631 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48883.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48883.log.gz 2024-05-17T19:32:27.633 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52904.log 2024-05-17T19:32:27.633 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.56439.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56439.log.gz 2024-05-17T19:32:27.633 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59713.log 2024-05-17T19:32:27.634 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.52904.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52904.log.gz 2024-05-17T19:32:27.636 INFO:teuthology.orchestra.run.smithi195.stderr:gzip 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43548.log.gz 2024-05-17T19:32:27.636 INFO:teuthology.orchestra.run.smithi195.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45631.log 2024-05-17T19:32:27.636 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43485.log 2024-05-17T19:32:27.636 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59713.log: /var/log/ceph/cluster2-client.mirror.45631.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59713.log.gz 2024-05-17T19:32:27.637 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54363.log 2024-05-17T19:32:27.641 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.43485.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45631.log.gz 2024-05-17T19:32:27.641 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.44822.log 2024-05-17T19:32:27.642 INFO:teuthology.orchestra.run.smithi195.stderr: 93.3%/var/log/ceph/cluster1-client.mirror.54363.log: -- replaced with /var/log/ceph/cluster2-client.mirror.43485.log.gz 2024-05-17T19:32:27.642 INFO:teuthology.orchestra.run.smithi195.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54363.log.gz 2024-05-17T19:32:27.646 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41861.log 2024-05-17T19:32:27.646 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.1.44822.log: --verbose -- /var/log/ceph/cluster2-client.mirror.51169.log 2024-05-17T19:32:27.647 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.41861.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41861.log.gz 2024-05-17T19:32:27.647 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47881.log 2024-05-17T19:32:27.647 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51169.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51169.log.gz 2024-05-17T19:32:27.647 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50658.log 2024-05-17T19:32:27.647 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47881.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47881.log.gz 2024-05-17T19:32:27.648 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51398.log 2024-05-17T19:32:27.651 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50658.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51880.log.gz 2024-05-17T19:32:27.652 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59166.log 2024-05-17T19:32:27.652 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51398.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51398.log.gz 2024-05-17T19:32:27.654 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62033.log 2024-05-17T19:32:27.654 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.59166.log: 0.0% 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.1.44822.log.gz 2024-05-17T19:32:27.654 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.59166.log.gz 2024-05-17T19:32:27.654 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47832.log 2024-05-17T19:32:27.655 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47497.log 2024-05-17T19:32:27.656 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62033.log: /var/log/ceph/cluster2-client.mirror.47832.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62033.log.gz 2024-05-17T19:32:27.656 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53065.log 2024-05-17T19:32:27.656 INFO:teuthology.orchestra.run.smithi195.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47832.log.gz 2024-05-17T19:32:27.656 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63303.log 2024-05-17T19:32:27.657 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.53065.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53065.log.gz 2024-05-17T19:32:27.659 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61633.log 2024-05-17T19:32:27.659 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63303.log: 10.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63303.log.gz 2024-05-17T19:32:27.659 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51965.log 2024-05-17T19:32:27.661 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61633.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61633.log.gz 2024-05-17T19:32:27.661 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40898.log 2024-05-17T19:32:27.661 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.51965.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51965.log.gz 2024-05-17T19:32:27.662 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48636.log 2024-05-17T19:32:27.667 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.40898.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40898.log.gz 2024-05-17T19:32:27.669 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63024.log 2024-05-17T19:32:27.669 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48636.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48636.log.gz 2024-05-17T19:32:27.670 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54481.log 2024-05-17T19:32:27.670 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63024.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63024.log.gz 2024-05-17T19:32:27.671 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56784.log 2024-05-17T19:32:27.684 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.54481.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46909.log 2024-05-17T19:32:27.685 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56784.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56784.log.gz 2024-05-17T19:32:27.693 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60233.log 2024-05-17T19:32:27.694 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46909.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46909.log.gz 2024-05-17T19:32:27.699 INFO:teuthology.orchestra.run.smithi195.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54481.log.gz 2024-05-17T19:32:27.700 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46158.log 2024-05-17T19:32:27.701 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60233.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60233.log.gz 2024-05-17T19:32:27.701 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41672.log 2024-05-17T19:32:27.704 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.46158.log: 94.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46158.log.gz 2024-05-17T19:32:27.704 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55196.log 2024-05-17T19:32:27.705 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.41672.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41672.log.gz 2024-05-17T19:32:27.705 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59493.log 2024-05-17T19:32:27.706 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55196.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55196.log.gz 2024-05-17T19:32:27.706 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40610.log 2024-05-17T19:32:27.707 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59493.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59493.log.gz 2024-05-17T19:32:27.708 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59433.log 2024-05-17T19:32:27.708 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.40610.log: 73.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40610.log.gz 2024-05-17T19:32:27.709 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54404.log 2024-05-17T19:32:27.709 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59433.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59433.log.gz 2024-05-17T19:32:27.710 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60053.log 2024-05-17T19:32:27.711 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54404.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54404.log.gz 2024-05-17T19:32:27.712 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42002.log 2024-05-17T19:32:27.712 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60053.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60053.log.gz 2024-05-17T19:32:27.713 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60913.log 2024-05-17T19:32:27.719 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.42002.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42002.log.gz 2024-05-17T19:32:27.719 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46367.log 2024-05-17T19:32:27.720 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60913.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60913.log.gz 2024-05-17T19:32:27.720 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58993.log 2024-05-17T19:32:27.721 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46367.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46367.log.gz 2024-05-17T19:32:27.722 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46287.log 2024-05-17T19:32:27.722 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58993.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58993.log.gz 2024-05-17T19:32:27.723 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42512.log 2024-05-17T19:32:27.724 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46287.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46287.log.gz 2024-05-17T19:32:27.724 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62990.log 2024-05-17T19:32:27.730 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.42512.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42512.log.gz 2024-05-17T19:32:27.730 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54743.log 2024-05-17T19:32:27.731 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.62990.log: 96.3% 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62990.log.gz 2024-05-17T19:32:27.731 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.3.57139.log.gz 2024-05-17T19:32:27.731 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57025.log 2024-05-17T19:32:27.732 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57757.log 2024-05-17T19:32:27.732 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.54743.log: /var/log/ceph/cluster1-client.mirror.57025.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57025.log.gz 2024-05-17T19:32:27.733 INFO:teuthology.orchestra.run.smithi195.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54743.log.gz 2024-05-17T19:32:27.733 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50836.log 2024-05-17T19:32:27.734 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38540.log 2024-05-17T19:32:27.734 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.57757.log: /var/log/ceph/cluster2-client.mirror.50836.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50836.log.gz 2024-05-17T19:32:27.735 INFO:teuthology.orchestra.run.smithi195.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57757.log.gz 2024-05-17T19:32:27.735 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51514.log 2024-05-17T19:32:27.735 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59393.log 2024-05-17T19:32:27.736 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.38540.log: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.38540.log.gz 2024-05-17T19:32:27.736 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62173.log 2024-05-17T19:32:27.736 INFO:teuthology.orchestra.run.smithi195.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51514.log.gz 2024-05-17T19:32:27.737 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59393.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59393.log.gz 2024-05-17T19:32:27.743 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63503.log 2024-05-17T19:32:27.743 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45131.log 2024-05-17T19:32:27.743 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62173.log: 93.8%/var/log/ceph/cluster2-client.admin.63503.log: -- replaced with /var/log/ceph/cluster2-client.mirror.48131.log.gz 2024-05-17T19:32:27.744 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63503.log.gz 2024-05-17T19:32:27.744 INFO:teuthology.orchestra.run.smithi195.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62173.log.gz 2024-05-17T19:32:27.744 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60033.log 2024-05-17T19:32:27.744 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52820.log 2024-05-17T19:32:27.744 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63171.log 2024-05-17T19:32:27.744 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60033.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45131.log.gz 2024-05-17T19:32:27.744 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.52820.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60033.log.gz 2024-05-17T19:32:27.744 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62013.log 2024-05-17T19:32:27.744 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63171.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63171.log.gz 2024-05-17T19:32:27.745 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63100.log 2024-05-17T19:32:27.745 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61013.log 2024-05-17T19:32:27.745 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63100.log: /var/log/ceph/cluster1-client.mirror.62013.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52820.log.gz 2024-05-17T19:32:27.745 INFO:teuthology.orchestra.run.smithi195.stderr: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63100.log.gz 2024-05-17T19:32:27.745 INFO:teuthology.orchestra.run.smithi195.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62013.log.gz 2024-05-17T19:32:27.745 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52430.log 2024-05-17T19:32:27.745 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61013.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58478.log 2024-05-17T19:32:27.745 INFO:teuthology.orchestra.run.smithi195.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61013.log.gz 2024-05-17T19:32:27.745 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.52430.log: -- /var/log/ceph/cluster1-client.admin.42157.log 2024-05-17T19:32:27.746 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58478.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52430.log.gz 2024-05-17T19:32:27.746 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61693.log 2024-05-17T19:32:27.746 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.42157.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.42157.log.gz 2024-05-17T19:32:27.751 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58851.log 2024-05-17T19:32:27.751 INFO:teuthology.orchestra.run.smithi195.stderr: 92.3%/var/log/ceph/cluster1-client.mirror.61693.log: -- replaced with /var/log/ceph/cluster1-client.mirror.58478.log.gz 2024-05-17T19:32:27.751 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44885.log 2024-05-17T19:32:27.751 INFO:teuthology.orchestra.run.smithi195.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61693.log.gz 2024-05-17T19:32:27.752 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54384.log 2024-05-17T19:32:27.752 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.58851.log: gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.44885.log: -- /var/log/ceph/cluster2-client.mirror.51652.log 2024-05-17T19:32:27.752 INFO:teuthology.orchestra.run.smithi195.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.44885.log.gz 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58851.log.gz 2024-05-17T19:32:27.752 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-05-17T19:32:27.752 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.54384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55747.log 2024-05-17T19:32:27.752 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51652.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33414.log 2024-05-17T19:32:27.752 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55747.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51652.log.gz 2024-05-17T19:32:27.752 INFO:teuthology.orchestra.run.smithi195.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54384.log.gz 2024-05-17T19:32:27.752 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54143.log 2024-05-17T19:32:27.753 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.33414.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47083.log 2024-05-17T19:32:27.753 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33414.log.gz 2024-05-17T19:32:27.753 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.54143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58229.log 2024-05-17T19:32:27.753 INFO:teuthology.orchestra.run.smithi195.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55747.log.gz 2024-05-17T19:32:27.753 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47083.log: 80.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55110.log 2024-05-17T19:32:27.753 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47083.log.gz 2024-05-17T19:32:27.753 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58229.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54143.log.gz 2024-05-17T19:32:27.753 INFO:teuthology.orchestra.run.smithi195.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58229.log.gz 2024-05-17T19:32:27.763 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45221.log 2024-05-17T19:32:27.763 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.55110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42375.log 2024-05-17T19:32:27.763 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59633.log 2024-05-17T19:32:27.763 INFO:teuthology.orchestra.run.smithi195.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55110.log.gz 2024-05-17T19:32:27.763 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.42375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41933.log 2024-05-17T19:32:27.763 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59633.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42375.log.gz 2024-05-17T19:32:27.763 INFO:teuthology.orchestra.run.smithi195.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47986.log 2024-05-17T19:32:27.764 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59633.log.gz 2024-05-17T19:32:27.764 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51858.log 2024-05-17T19:32:27.764 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.41933.log: /var/log/ceph/cluster2-client.mirror.47986.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47986.log.gz 2024-05-17T19:32:27.764 INFO:teuthology.orchestra.run.smithi195.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41933.log.gz 2024-05-17T19:32:27.764 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48925.log 2024-05-17T19:32:27.764 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51858.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58048.log 2024-05-17T19:32:27.764 INFO:teuthology.orchestra.run.smithi195.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51858.log.gz 2024-05-17T19:32:27.764 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48925.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48925.log.gz 2024-05-17T19:32:27.765 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53165.log 2024-05-17T19:32:27.765 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.58048.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49221.log 2024-05-17T19:32:27.765 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.53165.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45221.log.gz 89.7% 2024-05-17T19:32:27.765 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.58048.log.gz 2024-05-17T19:32:27.765 INFO:teuthology.orchestra.run.smithi195.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53165.log.gz 2024-05-17T19:32:27.765 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60293.log 2024-05-17T19:32:27.765 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63415.log 2024-05-17T19:32:27.765 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60293.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55047.log 2024-05-17T19:32:27.765 INFO:teuthology.orchestra.run.smithi195.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60293.log.gz 2024-05-17T19:32:27.766 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.63415.log: 6.1% -- replaced with /var/log/ceph/cluster1-client.admin.63415.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60193.log 2024-05-17T19:32:27.766 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-05-17T19:32:27.766 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55047.log: gzip -5 --verbose 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55047.log.gz 2024-05-17T19:32:27.766 INFO:teuthology.orchestra.run.smithi195.stderr: -- /var/log/ceph/cluster1-client.mirror.1.49258.log 2024-05-17T19:32:27.766 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.38071.log 2024-05-17T19:32:27.766 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60193.log: /var/log/ceph/cluster1-client.mirror.1.49258.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60193.log.gz 2024-05-17T19:32:27.766 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54002.log 2024-05-17T19:32:27.770 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.2.38071.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48312.log.gz 2024-05-17T19:32:27.771 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.48547.log 2024-05-17T19:32:27.772 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.54002.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54002.log.gz 2024-05-17T19:32:27.773 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52261.log 2024-05-17T19:32:27.786 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.3.48547.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45952.log 2024-05-17T19:32:27.787 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52261.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52261.log.gz 2024-05-17T19:32:27.793 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52350.log 2024-05-17T19:32:27.799 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45952.log: 93.7% 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.49258.log.gz 2024-05-17T19:32:27.800 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.3.48547.log.gz 2024-05-17T19:32:27.800 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61573.log 2024-05-17T19:32:27.801 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52350.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.52350.log.gz 2024-05-17T19:32:27.802 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46658.log 2024-05-17T19:32:27.803 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61573.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61573.log.gz 2024-05-17T19:32:27.803 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59953.log 2024-05-17T19:32:27.805 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46658.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46658.log.gz 2024-05-17T19:32:27.805 INFO:teuthology.orchestra.run.smithi195.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45952.log.gz 2024-05-17T19:32:27.805 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49384.log 2024-05-17T19:32:27.806 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38492.log 2024-05-17T19:32:27.806 INFO:teuthology.orchestra.run.smithi195.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59953.log.gz 2024-05-17T19:32:27.807 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49384.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49384.log.gz 2024-05-17T19:32:27.807 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59593.log 2024-05-17T19:32:27.807 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43657.log 2024-05-17T19:32:27.807 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.38492.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38492.log.gz 2024-05-17T19:32:27.807 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59593.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-05-17T19:32:27.808 INFO:teuthology.orchestra.run.smithi195.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59593.log.gz 2024-05-17T19:32:27.808 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.43657.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.43657.log.gz 2024-05-17T19:32:27.808 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62556.log 2024-05-17T19:32:27.809 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54163.log 2024-05-17T19:32:27.809 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-mon.a.log: /var/log/ceph/cluster1-client.mirror.62556.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62556.log.gz 2024-05-17T19:32:27.821 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42447.log 2024-05-17T19:32:27.823 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54163.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54163.log.gz 2024-05-17T19:32:27.834 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55289.log 2024-05-17T19:32:27.835 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.42447.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42447.log.gz 2024-05-17T19:32:27.836 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43528.log 2024-05-17T19:32:27.836 INFO:teuthology.orchestra.run.smithi195.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.2.38071.log.gz 2024-05-17T19:32:27.837 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.55289.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55289.log.gz 2024-05-17T19:32:27.838 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55819.log 2024-05-17T19:32:27.838 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40354.log 2024-05-17T19:32:27.840 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.43528.log: /var/log/ceph/cluster2-client.mirror.55819.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55819.log.gz 2024-05-17T19:32:27.844 INFO:teuthology.orchestra.run.smithi195.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43528.log.gz 2024-05-17T19:32:27.844 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61673.log 2024-05-17T19:32:27.844 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49004.log 2024-05-17T19:32:27.845 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.40354.log: /var/log/ceph/cluster1-client.mirror.61673.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61673.log.gz 2024-05-17T19:32:27.846 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49659.log.gz 2024-05-17T19:32:27.846 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38171.log 2024-05-17T19:32:27.848 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49004.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51694.log 2024-05-17T19:32:27.848 INFO:teuthology.orchestra.run.smithi195.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49004.log.gz 2024-05-17T19:32:27.848 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.38171.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.38171.log.gz 2024-05-17T19:32:27.848 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57449.log 2024-05-17T19:32:27.848 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51694.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45459.log 2024-05-17T19:32:27.849 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57449.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57449.log.gz 2024-05-17T19:32:27.849 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62518.log 2024-05-17T19:32:27.850 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45459.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40354.log.gz 2024-05-17T19:32:27.854 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57401.log 2024-05-17T19:32:27.854 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62518.log: 68.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62518.log.gz 2024-05-17T19:32:27.856 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56808.log 2024-05-17T19:32:27.856 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57401.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45459.log.gz 2024-05-17T19:32:27.856 INFO:teuthology.orchestra.run.smithi195.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57401.log.gz 2024-05-17T19:32:27.857 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45545.log 2024-05-17T19:32:27.857 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56808.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63068.log 2024-05-17T19:32:27.857 INFO:teuthology.orchestra.run.smithi195.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56808.log.gz 2024-05-17T19:32:27.859 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45545.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57827.log 2024-05-17T19:32:27.859 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63068.log: 6.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63068.log.gz 2024-05-17T19:32:27.860 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52237.log 2024-05-17T19:32:27.860 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.57827.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57827.log.gz 2024-05-17T19:32:27.860 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50293.log 2024-05-17T19:32:27.860 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52237.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52237.log.gz 2024-05-17T19:32:27.862 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42220.log 2024-05-17T19:32:27.862 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50293.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50293.log.gz 2024-05-17T19:32:27.862 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-05-17T19:32:27.863 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.42220.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45545.log.gz 2024-05-17T19:32:27.864 INFO:teuthology.orchestra.run.smithi195.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42220.log.gz 2024-05-17T19:32:27.864 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45909.log 2024-05-17T19:32:27.864 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62910.log 2024-05-17T19:32:27.869 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45909.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45909.log.gz 2024-05-17T19:32:27.869 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43111.log 2024-05-17T19:32:27.870 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.62910.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.admin.62910.log.gz 2024-05-17T19:32:27.870 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53306.log 2024-05-17T19:32:27.871 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.43111.log: 82.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43111.log.gz 2024-05-17T19:32:27.871 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50974.log 2024-05-17T19:32:27.873 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.53306.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53306.log.gz 2024-05-17T19:32:27.873 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61033.log 2024-05-17T19:32:27.888 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50974.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60213.log 2024-05-17T19:32:27.889 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61033.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61033.log.gz 2024-05-17T19:32:27.904 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50167.log 2024-05-17T19:32:27.905 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60213.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60213.log.gz 2024-05-17T19:32:27.917 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60173.log 2024-05-17T19:32:27.918 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50167.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50167.log.gz 2024-05-17T19:32:27.937 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63292.log 2024-05-17T19:32:27.937 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60173.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60173.log.gz 2024-05-17T19:32:27.947 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49120.log 2024-05-17T19:32:27.947 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63292.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63292.log.gz 2024-05-17T19:32:27.956 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.50204.log 2024-05-17T19:32:27.958 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49120.log: 86.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49120.log.gz 2024-05-17T19:32:27.970 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45307.log 2024-05-17T19:32:27.979 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.2.50204.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.51732.log 2024-05-17T19:32:27.988 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45307.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47918.log 2024-05-17T19:32:27.998 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.3.51732.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45307.log.gz 2024-05-17T19:32:27.998 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44574.log 2024-05-17T19:32:27.999 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47918.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47918.log.gz 2024-05-17T19:32:28.010 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42130.log 2024-05-17T19:32:28.013 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.44574.log: 95.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44574.log.gz 2024-05-17T19:32:28.021 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47775.log.gz 2024-05-17T19:32:28.032 INFO:teuthology.orchestra.run.smithi134.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-05-17T19:32:28.033 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53876.log 2024-05-17T19:32:28.034 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.42130.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.42130.log.gz 2024-05-17T19:32:28.055 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47497.log.gz 2024-05-17T19:32:28.055 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51152.log 2024-05-17T19:32:28.056 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.53876.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53876.log.gz 2024-05-17T19:32:28.057 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42045.log 2024-05-17T19:32:28.058 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51152.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51152.log.gz 2024-05-17T19:32:28.058 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56377.log 2024-05-17T19:32:28.064 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.42045.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42045.log.gz 2024-05-17T19:32:28.064 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58389.log 2024-05-17T19:32:28.064 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.56377.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56377.log.gz 2024-05-17T19:32:28.066 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63403.log 2024-05-17T19:32:28.066 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58389.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58389.log.gz 2024-05-17T19:32:28.068 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38516.log 2024-05-17T19:32:28.068 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.63403.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.admin.63403.log.gz 2024-05-17T19:32:28.069 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54785.log 2024-05-17T19:32:28.069 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.38516.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38516.log.gz 2024-05-17T19:32:28.069 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38710.log 2024-05-17T19:32:28.070 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.54785.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54785.log.gz 2024-05-17T19:32:28.070 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60333.log 2024-05-17T19:32:28.070 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.38710.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38710.log.gz 2024-05-17T19:32:28.072 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-05-17T19:32:28.072 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60333.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60333.log.gz 2024-05-17T19:32:28.073 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.46158.log 2024-05-17T19:32:28.074 INFO:teuthology.orchestra.run.smithi195.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.3.51732.log.gz 2024-05-17T19:32:28.074 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51556.log 2024-05-17T19:32:28.078 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.0.46158.log: 95.1% -- replaced with /var/log/ceph/cluster2-client.mirror.2.50204.log.gz 2024-05-17T19:32:28.086 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52775.log 2024-05-17T19:32:28.087 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51556.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51556.log.gz 2024-05-17T19:32:28.100 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52703.log 2024-05-17T19:32:28.101 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52775.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52775.log.gz 2024-05-17T19:32:28.101 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50737.log 2024-05-17T19:32:28.102 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52703.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52703.log.gz 2024-05-17T19:32:28.115 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47940.log 2024-05-17T19:32:28.125 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50737.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59773.log 2024-05-17T19:32:28.137 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47940.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60593.log 2024-05-17T19:32:28.137 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.0.46158.log.gz 2024-05-17T19:32:28.138 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59773.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59773.log.gz 2024-05-17T19:32:28.150 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51435.log 2024-05-17T19:32:28.150 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60593.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60593.log.gz 2024-05-17T19:32:28.173 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52526.log 2024-05-17T19:32:28.173 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51435.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51435.log.gz 2024-05-17T19:32:28.183 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40717.log 2024-05-17T19:32:28.184 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52526.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52526.log.gz 2024-05-17T19:32:28.192 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51090.log 2024-05-17T19:32:28.206 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.40717.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40717.log.gz 2024-05-17T19:32:28.211 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.56225.log 2024-05-17T19:32:28.212 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51090.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51090.log.gz 2024-05-17T19:32:28.227 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44123.log 2024-05-17T19:32:28.240 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.0.56225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60653.log 2024-05-17T19:32:28.242 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.44123.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44123.log.gz 2024-05-17T19:32:28.249 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51268.log 2024-05-17T19:32:28.250 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60653.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60653.log.gz 2024-05-17T19:32:28.251 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50498.log 2024-05-17T19:32:28.251 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51268.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51268.log.gz 2024-05-17T19:32:28.262 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59933.log 2024-05-17T19:32:28.268 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56184.log 2024-05-17T19:32:28.269 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59933.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59933.log.gz 2024-05-17T19:32:28.282 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.42220.log 2024-05-17T19:32:28.283 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56184.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56184.log.gz 2024-05-17T19:32:28.289 INFO:teuthology.orchestra.run.smithi195.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.56225.log.gz 2024-05-17T19:32:28.293 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55839.log 2024-05-17T19:32:28.304 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.2.42220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47534.log 2024-05-17T19:32:28.305 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.55839.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55839.log.gz 2024-05-17T19:32:28.317 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44437.log 2024-05-17T19:32:28.317 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.47534.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.47534.log.gz 2024-05-17T19:32:28.340 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.2.42220.log.gz 2024-05-17T19:32:28.341 INFO:teuthology.orchestra.run.smithi195.stderr: /var/log/ceph/cluster1-client.mirror.57066.log 2024-05-17T19:32:28.350 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.44437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46038.log 2024-05-17T19:32:28.352 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57066.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57066.log.gz 2024-05-17T19:32:28.364 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50658.log.gz 2024-05-17T19:32:28.365 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62466.log 2024-05-17T19:32:28.367 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.46038.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50974.log.gz 2024-05-17T19:32:28.367 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60853.log 2024-05-17T19:32:28.369 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62466.log: 68.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62466.log.gz 2024-05-17T19:32:28.369 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59613.log 2024-05-17T19:32:28.371 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60853.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60853.log.gz 2024-05-17T19:32:28.371 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50636.log 2024-05-17T19:32:28.371 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59613.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59613.log.gz 2024-05-17T19:32:28.371 INFO:teuthology.orchestra.run.smithi195.stderr: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55461.log 2024-05-17T19:32:28.371 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.46038.log.gz 2024-05-17T19:32:28.371 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50636.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49142.log 2024-05-17T19:32:28.374 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55461.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50636.log.gz 2024-05-17T19:32:28.374 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53486.log 2024-05-17T19:32:28.374 INFO:teuthology.orchestra.run.smithi195.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55461.log.gz 2024-05-17T19:32:28.374 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49142.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62646.log 2024-05-17T19:32:28.374 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53486.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53486.log.gz 2024-05-17T19:32:28.375 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61093.log 2024-05-17T19:32:28.375 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62646.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62646.log.gz 2024-05-17T19:32:28.377 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46946.log 2024-05-17T19:32:28.377 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61093.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61093.log.gz 2024-05-17T19:32:28.377 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58969.log 2024-05-17T19:32:28.377 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46946.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46946.log.gz 2024-05-17T19:32:28.378 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40519.log 2024-05-17T19:32:28.379 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58969.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58969.log.gz 2024-05-17T19:32:28.379 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53213.log 2024-05-17T19:32:28.379 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.40519.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.40519.log.gz 2024-05-17T19:32:28.380 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61053.log 2024-05-17T19:32:28.381 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53213.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53213.log.gz 2024-05-17T19:32:28.381 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47795.log 2024-05-17T19:32:28.383 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61053.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61053.log.gz 2024-05-17T19:32:28.383 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63080.log 2024-05-17T19:32:28.384 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47795.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47795.log.gz 2024-05-17T19:32:28.384 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53597.log 2024-05-17T19:32:28.385 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63080.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63080.log.gz 2024-05-17T19:32:28.385 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54722.log 2024-05-17T19:32:28.386 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.53597.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53597.log.gz 2024-05-17T19:32:28.387 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63429.log 2024-05-17T19:32:28.388 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54722.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54722.log.gz 2024-05-17T19:32:28.389 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63060.log 2024-05-17T19:32:28.389 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.63429.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.63429.log.gz 2024-05-17T19:32:28.391 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60433.log 2024-05-17T19:32:28.391 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63060.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63060.log.gz 2024-05-17T19:32:28.391 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59753.log 2024-05-17T19:32:28.392 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60433.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60433.log.gz 2024-05-17T19:32:28.392 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59653.log 2024-05-17T19:32:28.393 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59753.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59753.log.gz 2024-05-17T19:32:28.394 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53261.log 2024-05-17T19:32:28.395 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59653.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59653.log.gz 2024-05-17T19:32:28.395 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47372.log 2024-05-17T19:32:28.396 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53261.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53261.log.gz 2024-05-17T19:32:28.397 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53189.log 2024-05-17T19:32:28.399 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47372.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47372.log.gz 2024-05-17T19:32:28.399 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45889.log 2024-05-17T19:32:28.400 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53189.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53189.log.gz 2024-05-17T19:32:28.400 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61953.log 2024-05-17T19:32:28.406 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45889.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45889.log.gz 2024-05-17T19:32:28.407 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48653.log 2024-05-17T19:32:28.407 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61953.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61953.log.gz 2024-05-17T19:32:28.408 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46757.log 2024-05-17T19:32:28.410 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48653.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48653.log.gz 2024-05-17T19:32:28.410 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.47179.log 2024-05-17T19:32:28.412 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46757.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46757.log.gz 2024-05-17T19:32:28.412 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42065.log 2024-05-17T19:32:28.425 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.2.47179.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37195.log 2024-05-17T19:32:28.432 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.42065.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41885.log 2024-05-17T19:32:28.439 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.37195.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44261.log 2024-05-17T19:32:28.439 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37195.log.gz 2024-05-17T19:32:28.442 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.41885.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.41885.log.gz 2024-05-17T19:32:28.444 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42065.log.gz 2024-05-17T19:32:28.457 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42833.log 2024-05-17T19:32:28.457 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.44261.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44261.log.gz 2024-05-17T19:32:28.468 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40737.log 2024-05-17T19:32:28.468 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.42833.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42833.log.gz 2024-05-17T19:32:28.491 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62962.log 2024-05-17T19:32:28.491 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.40737.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40737.log.gz 2024-05-17T19:32:28.507 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52502.log 2024-05-17T19:32:28.508 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.62962.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62962.log.gz 2024-05-17T19:32:28.527 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44935.log 2024-05-17T19:32:28.528 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52502.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52502.log.gz 2024-05-17T19:32:28.556 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62594.log 2024-05-17T19:32:28.557 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.44935.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44935.log.gz 2024-05-17T19:32:28.568 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40985.log 2024-05-17T19:32:28.568 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62594.log: 55.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62594.log.gz 2024-05-17T19:32:28.577 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44393.log 2024-05-17T19:32:28.579 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.40985.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49221.log.gz 2024-05-17T19:32:28.592 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56977.log 2024-05-17T19:32:28.608 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.44393.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43020.log 2024-05-17T19:32:28.608 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56977.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56977.log.gz 2024-05-17T19:32:28.611 INFO:teuthology.orchestra.run.smithi195.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44393.log.gz 2024-05-17T19:32:28.624 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49935.log 2024-05-17T19:32:28.625 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.43020.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.43020.log.gz 2024-05-17T19:32:28.638 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60893.log 2024-05-17T19:32:28.650 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49935.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62851.log 2024-05-17T19:32:28.651 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60893.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60893.log.gz 2024-05-17T19:32:28.659 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.58699.log 2024-05-17T19:32:28.660 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62851.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62851.log.gz 2024-05-17T19:32:28.668 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50737.log.gz 2024-05-17T19:32:28.669 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51694.log.gz 2024-05-17T19:32:28.674 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62389.log 2024-05-17T19:32:28.698 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.1.58699.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52410.log 2024-05-17T19:32:28.699 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.62389.log: 70.8% -- replaced with /var/log/ceph/cluster1-client.admin.62389.log.gz 2024-05-17T19:32:28.722 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39120.log 2024-05-17T19:32:28.723 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.52410.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52410.log.gz 2024-05-17T19:32:28.730 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50498.log.gz 2024-05-17T19:32:28.736 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54267.log 2024-05-17T19:32:28.736 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.39120.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.1.58699.log.gz 2024-05-17T19:32:28.736 INFO:teuthology.orchestra.run.smithi195.stderr: 56.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39120.log.gz 2024-05-17T19:32:28.747 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63045.log 2024-05-17T19:32:28.747 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54267.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54267.log.gz 2024-05-17T19:32:28.753 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49471.log 2024-05-17T19:32:28.766 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.63045.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.admin.63045.log.gz 2024-05-17T19:32:28.767 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46717.log 2024-05-17T19:32:28.780 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49471.log: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63108.log 2024-05-17T19:32:28.780 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49471.log.gz 2024-05-17T19:32:28.787 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46717.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49521.log 2024-05-17T19:32:28.799 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.63108.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58699.log 2024-05-17T19:32:28.800 INFO:teuthology.orchestra.run.smithi195.stderr: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.63108.log.gz 2024-05-17T19:32:28.800 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49521.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49521.log.gz 2024-05-17T19:32:28.808 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58586.log 2024-05-17T19:32:28.809 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58699.log: 84.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58699.log.gz 2024-05-17T19:32:28.821 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46527.log 2024-05-17T19:32:28.822 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58586.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58586.log.gz 2024-05-17T19:32:28.838 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40155.log 2024-05-17T19:32:28.839 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46527.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46527.log.gz 2024-05-17T19:32:28.852 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59893.log 2024-05-17T19:32:28.853 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.40155.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40155.log.gz 2024-05-17T19:32:28.871 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50476.log 2024-05-17T19:32:28.872 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59893.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59893.log.gz 2024-05-17T19:32:28.876 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49142.log.gz 2024-05-17T19:32:28.889 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49021.log 2024-05-17T19:32:28.890 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50476.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50476.log.gz 2024-05-17T19:32:28.901 INFO:teuthology.orchestra.run.smithi195.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-05-17T19:32:28.902 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53107.log 2024-05-17T19:32:28.904 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49021.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49021.log.gz 2024-05-17T19:32:28.904 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53414.log 2024-05-17T19:32:28.905 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.53107.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53107.log.gz 2024-05-17T19:32:28.906 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48253.log 2024-05-17T19:32:28.908 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.53414.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53414.log.gz 2024-05-17T19:32:28.908 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48942.log 2024-05-17T19:32:28.909 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48253.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48253.log.gz 2024-05-17T19:32:28.909 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42878.log 2024-05-17T19:32:28.910 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48942.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48942.log.gz 2024-05-17T19:32:28.911 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42351.log 2024-05-17T19:32:28.916 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.42878.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42878.log.gz 2024-05-17T19:32:28.917 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51902.log 2024-05-17T19:32:28.919 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.42351.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42351.log.gz 2024-05-17T19:32:28.919 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45846.log 2024-05-17T19:32:28.920 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51902.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51902.log.gz 2024-05-17T19:32:28.920 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57622.log 2024-05-17T19:32:28.923 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45846.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47940.log.gz 2024-05-17T19:32:28.925 INFO:teuthology.orchestra.run.smithi195.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45846.log.gz 2024-05-17T19:32:28.926 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58829.log 2024-05-17T19:32:28.927 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57622.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57622.log.gz 2024-05-17T19:32:28.927 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38002.log 2024-05-17T19:32:28.929 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58829.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58829.log.gz 2024-05-17T19:32:28.930 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63161.log 2024-05-17T19:32:28.943 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.38002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58610.log 2024-05-17T19:32:28.943 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63161.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63161.log.gz 2024-05-17T19:32:28.950 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55413.log 2024-05-17T19:32:28.950 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58610.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58610.log.gz 2024-05-17T19:32:28.958 INFO:teuthology.orchestra.run.smithi195.stderr: 94.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38002.log.gz 2024-05-17T19:32:28.959 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49451.log 2024-05-17T19:32:28.960 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55413.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55413.log.gz 2024-05-17T19:32:28.960 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47055.log 2024-05-17T19:32:28.961 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49451.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49451.log.gz 2024-05-17T19:32:28.962 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47305.log 2024-05-17T19:32:28.975 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47055.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43392.log 2024-05-17T19:32:28.976 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47305.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47305.log.gz 2024-05-17T19:32:28.989 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46267.log 2024-05-17T19:32:28.990 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.43392.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43392.log.gz 2024-05-17T19:32:29.003 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47013.log 2024-05-17T19:32:29.004 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46267.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46267.log.gz 2024-05-17T19:32:29.012 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39137.log 2024-05-17T19:32:29.013 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47013.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47013.log.gz 2024-05-17T19:32:29.022 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59238.log 2024-05-17T19:32:29.022 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.39137.log: 56.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39137.log.gz 2024-05-17T19:32:29.035 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40546.log 2024-05-17T19:32:29.048 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59238.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56088.log 2024-05-17T19:32:29.049 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.40546.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.40546.log.gz 2024-05-17T19:32:29.052 INFO:teuthology.orchestra.run.smithi195.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59238.log.gz 2024-05-17T19:32:29.058 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61293.log 2024-05-17T19:32:29.058 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.56088.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56088.log.gz 2024-05-17T19:32:29.067 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50757.log 2024-05-17T19:32:29.067 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61293.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61293.log.gz 2024-05-17T19:32:29.076 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42809.log 2024-05-17T19:32:29.076 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50757.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50757.log.gz 2024-05-17T19:32:29.076 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44437.log.gz 2024-05-17T19:32:29.077 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51494.log 2024-05-17T19:32:29.079 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.42809.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42809.log.gz 2024-05-17T19:32:29.079 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63389.log 2024-05-17T19:32:29.080 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51494.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51494.log.gz 2024-05-17T19:32:29.081 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54098.log 2024-05-17T19:32:29.082 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.63389.log: 9.2% -- replaced with /var/log/ceph/cluster1-client.admin.63389.log.gz 2024-05-17T19:32:29.082 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50100.log 2024-05-17T19:32:29.083 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54098.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54098.log.gz 2024-05-17T19:32:29.083 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51231.log 2024-05-17T19:32:29.085 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50100.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50100.log.gz 2024-05-17T19:32:29.085 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49434.log 2024-05-17T19:32:29.086 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51231.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51231.log.gz 2024-05-17T19:32:29.086 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59993.log 2024-05-17T19:32:29.087 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49434.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49434.log.gz 2024-05-17T19:32:29.088 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38642.log 2024-05-17T19:32:29.088 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59993.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59993.log.gz 2024-05-17T19:32:29.089 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60553.log 2024-05-17T19:32:29.090 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.38642.log: 57.1% -- replaced with /var/log/ceph/cluster1-client.mirror.38642.log.gz 2024-05-17T19:32:29.090 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40058.log 2024-05-17T19:32:29.090 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60553.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60553.log.gz 2024-05-17T19:32:29.092 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51457.log 2024-05-17T19:32:29.092 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.40058.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.40058.log.gz 2024-05-17T19:32:29.093 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60753.log 2024-05-17T19:32:29.106 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38762.log 2024-05-17T19:32:29.107 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60753.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60753.log.gz 2024-05-17T19:32:29.127 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63211.log 2024-05-17T19:32:29.128 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.38762.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.38762.log.gz 2024-05-17T19:32:29.140 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40833.log 2024-05-17T19:32:29.140 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63211.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63211.log.gz 2024-05-17T19:32:29.149 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42761.log 2024-05-17T19:32:29.150 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.40833.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40833.log.gz 2024-05-17T19:32:29.153 INFO:teuthology.orchestra.run.smithi195.stderr: 95.7% -- replaced with /var/log/ceph/cluster1-client.mirror.2.47179.log.gz 2024-05-17T19:32:29.154 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51369.log 2024-05-17T19:32:29.154 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.42761.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42761.log.gz 2024-05-17T19:32:29.155 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62273.log 2024-05-17T19:32:29.167 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51369.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50853.log 2024-05-17T19:32:29.168 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62273.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62273.log.gz 2024-05-17T19:32:29.178 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57947.log 2024-05-17T19:32:29.179 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50853.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50853.log.gz 2024-05-17T19:32:29.192 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42555.log 2024-05-17T19:32:29.193 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57947.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57947.log.gz 2024-05-17T19:32:29.199 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49493.log 2024-05-17T19:32:29.208 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.42555.log: 93.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46101.log 2024-05-17T19:32:29.213 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49493.log: -- replaced with /var/log/ceph/cluster1-client.mirror.42555.log.gz 2024-05-17T19:32:29.224 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52799.log 2024-05-17T19:32:29.225 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46101.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46101.log.gz 2024-05-17T19:32:29.232 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49935.log.gz 2024-05-17T19:32:29.234 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48905.log 2024-05-17T19:32:29.248 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52799.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52799.log.gz 2024-05-17T19:32:29.255 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42183.log 2024-05-17T19:32:29.275 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43896.log 2024-05-17T19:32:29.286 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.42183.log: 91.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49913.log 2024-05-17T19:32:29.286 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.43896.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43896.log.gz 2024-05-17T19:32:29.290 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.42183.log.gz 2024-05-17T19:32:29.294 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52751.log 2024-05-17T19:32:29.296 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49913.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47055.log.gz 2024-05-17T19:32:29.297 INFO:teuthology.orchestra.run.smithi195.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49913.log.gz 2024-05-17T19:32:29.305 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54763.log 2024-05-17T19:32:29.306 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52751.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52751.log.gz 2024-05-17T19:32:29.317 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63344.log 2024-05-17T19:32:29.319 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54763.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54763.log.gz 2024-05-17T19:32:29.320 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51457.log.gz 2024-05-17T19:32:29.322 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45393.log 2024-05-17T19:32:29.322 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63344.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63344.log.gz 2024-05-17T19:32:29.322 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46121.log 2024-05-17T19:32:29.328 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45393.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45393.log.gz 2024-05-17T19:32:29.328 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48023.log 2024-05-17T19:32:29.329 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46121.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46121.log.gz 2024-05-17T19:32:29.329 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51821.log 2024-05-17T19:32:29.332 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48023.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48023.log.gz 2024-05-17T19:32:29.332 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54927.log 2024-05-17T19:32:29.332 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51821.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51821.log.gz 2024-05-17T19:32:29.332 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45019.log 2024-05-17T19:32:29.332 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.54927.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54927.log.gz 2024-05-17T19:32:29.333 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52591.log 2024-05-17T19:32:29.339 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45019.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45019.log.gz 2024-05-17T19:32:29.339 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39210.log 2024-05-17T19:32:29.341 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52591.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52591.log.gz 2024-05-17T19:32:29.342 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57670.log 2024-05-17T19:32:29.343 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.39210.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39210.log.gz 2024-05-17T19:32:29.343 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60133.log 2024-05-17T19:32:29.344 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57670.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57670.log.gz 2024-05-17T19:32:29.344 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51347.log 2024-05-17T19:32:29.346 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60133.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60133.log.gz 2024-05-17T19:32:29.346 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51189.log 2024-05-17T19:32:29.347 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51347.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51347.log.gz 2024-05-17T19:32:29.347 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51132.log 2024-05-17T19:32:29.348 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51189.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51189.log.gz 2024-05-17T19:32:29.349 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43776.log 2024-05-17T19:32:29.363 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46407.log 2024-05-17T19:32:29.363 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.43776.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43776.log.gz 2024-05-17T19:32:29.377 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39909.log 2024-05-17T19:32:29.378 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46407.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46407.log.gz 2024-05-17T19:32:29.378 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58297.log 2024-05-17T19:32:29.399 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.39909.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59553.log 2024-05-17T19:32:29.406 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.58297.log: 89.7% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40985.log.gz 2024-05-17T19:32:29.406 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.58297.log.gz 2024-05-17T19:32:29.406 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46717.log.gz 2024-05-17T19:32:29.413 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62233.log 2024-05-17T19:32:29.413 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46327.log 2024-05-17T19:32:29.413 INFO:teuthology.orchestra.run.smithi195.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59553.log.gz 2024-05-17T19:32:29.414 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62233.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62233.log.gz 2024-05-17T19:32:29.415 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43710.log 2024-05-17T19:32:29.416 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46327.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46327.log.gz 2024-05-17T19:32:29.416 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41773.log 2024-05-17T19:32:29.417 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.43710.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43710.log.gz 2024-05-17T19:32:29.419 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62535.log 2024-05-17T19:32:29.419 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.41773.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.41773.log.gz 2024-05-17T19:32:29.421 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46547.log 2024-05-17T19:32:29.421 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62535.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62535.log.gz 2024-05-17T19:32:29.421 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52285.log 2024-05-17T19:32:29.422 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46547.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46547.log.gz 2024-05-17T19:32:29.423 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40036.log 2024-05-17T19:32:29.424 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52285.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52285.log.gz 2024-05-17T19:32:29.424 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55935.log 2024-05-17T19:32:29.425 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.40036.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.40036.log.gz 2024-05-17T19:32:29.425 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43684.log 2024-05-17T19:32:29.427 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55935.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55935.log.gz 2024-05-17T19:32:29.427 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53237.log 2024-05-17T19:32:29.428 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.43684.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.43684.log.gz 2024-05-17T19:32:29.428 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58809.log 2024-05-17T19:32:29.429 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53237.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53237.log.gz 2024-05-17T19:32:29.429 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.43111.log 2024-05-17T19:32:29.432 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.58809.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58809.log.gz 2024-05-17T19:32:29.432 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47414.log 2024-05-17T19:32:29.436 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.1.43111.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.43111.log.gz 2024-05-17T19:32:29.436 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62833.log 2024-05-17T19:32:29.450 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47414.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57851.log 2024-05-17T19:32:29.450 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.62833.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.admin.62833.log.gz 2024-05-17T19:32:29.463 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57088.log 2024-05-17T19:32:29.464 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57851.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57851.log.gz 2024-05-17T19:32:29.475 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55682.log 2024-05-17T19:32:29.476 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.57088.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57088.log.gz 2024-05-17T19:32:29.495 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53796.log 2024-05-17T19:32:29.496 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55682.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55682.log.gz 2024-05-17T19:32:29.505 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45522.log 2024-05-17T19:32:29.507 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.53796.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53796.log.gz 2024-05-17T19:32:29.522 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63200.log 2024-05-17T19:32:29.527 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45522.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45522.log.gz 2024-05-17T19:32:29.537 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52372.log 2024-05-17T19:32:29.538 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63200.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63200.log.gz 2024-05-17T19:32:29.546 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53635.log 2024-05-17T19:32:29.547 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.52372.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52372.log.gz 2024-05-17T19:32:29.565 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42331.log 2024-05-17T19:32:29.565 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.53635.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53635.log.gz 2024-05-17T19:32:29.577 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56688.log 2024-05-17T19:32:29.578 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.42331.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42331.log.gz 2024-05-17T19:32:29.589 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51290.log 2024-05-17T19:32:29.589 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.56688.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56688.log.gz 2024-05-17T19:32:29.609 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48863.log 2024-05-17T19:32:29.619 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61513.log 2024-05-17T19:32:29.619 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48863.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48863.log.gz 2024-05-17T19:32:29.627 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59453.log 2024-05-17T19:32:29.628 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61513.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61513.log.gz 2024-05-17T19:32:29.636 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51132.log.gz 2024-05-17T19:32:29.648 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56355.log 2024-05-17T19:32:29.649 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59453.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59453.log.gz 2024-05-17T19:32:29.662 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60253.log 2024-05-17T19:32:29.664 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56355.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56355.log.gz 2024-05-17T19:32:29.676 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55023.log 2024-05-17T19:32:29.677 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60253.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60253.log.gz 2024-05-17T19:32:29.690 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63231.log 2024-05-17T19:32:29.691 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55023.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55023.log.gz 2024-05-17T19:32:29.703 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51011.log 2024-05-17T19:32:29.703 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63231.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63231.log.gz 2024-05-17T19:32:29.712 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61493.log 2024-05-17T19:32:29.724 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51011.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63324.log 2024-05-17T19:32:29.725 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61493.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61493.log.gz 2024-05-17T19:32:29.725 INFO:teuthology.orchestra.run.smithi195.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51011.log.gz 2024-05-17T19:32:29.735 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39171.log 2024-05-17T19:32:29.735 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63324.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63324.log.gz 2024-05-17T19:32:29.755 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48072.log 2024-05-17T19:32:29.755 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.39171.log: 53.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39171.log.gz 2024-05-17T19:32:29.766 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57694.log 2024-05-17T19:32:29.766 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48072.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48072.log.gz 2024-05-17T19:32:29.782 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51369.log.gz 2024-05-17T19:32:29.784 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54626.log 2024-05-17T19:32:29.784 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57694.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57694.log.gz 2024-05-17T19:32:29.788 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60353.log 2024-05-17T19:32:29.788 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54626.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54626.log.gz 2024-05-17T19:32:29.788 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58658.log 2024-05-17T19:32:29.788 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60353.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60353.log.gz 2024-05-17T19:32:29.789 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43324.log 2024-05-17T19:32:29.789 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58658.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58658.log.gz 2024-05-17T19:32:29.789 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63258.log 2024-05-17T19:32:29.791 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.43324.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43324.log.gz 2024-05-17T19:32:29.791 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42492.log 2024-05-17T19:32:29.791 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63258.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63258.log.gz 2024-05-17T19:32:29.792 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37194.log 2024-05-17T19:32:29.794 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.42492.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48905.log.gz 2024-05-17T19:32:29.794 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57139.log 2024-05-17T19:32:29.794 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.37194.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37194.log.gz 2024-05-17T19:32:29.794 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47854.log 2024-05-17T19:32:29.796 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.57139.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42492.log.gz 2024-05-17T19:32:29.797 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56829.log 2024-05-17T19:32:29.803 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47854.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57139.log.gz 2024-05-17T19:32:29.804 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56160.log 2024-05-17T19:32:29.807 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.56829.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56829.log.gz 2024-05-17T19:32:29.807 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40418.log 2024-05-17T19:32:29.807 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56160.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56160.log.gz 2024-05-17T19:32:29.807 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57968.log 2024-05-17T19:32:29.819 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.40418.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53534.log 2024-05-17T19:32:29.821 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.57968.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57968.log.gz 2024-05-17T19:32:29.834 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44097.log 2024-05-17T19:32:29.834 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53534.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53534.log.gz 2024-05-17T19:32:29.848 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49493.log.gz 2024-05-17T19:32:29.849 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41957.log 2024-05-17T19:32:29.850 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.44097.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.44097.log.gz 2024-05-17T19:32:29.852 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36995.log 2024-05-17T19:32:29.852 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.41957.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41957.log.gz 2024-05-17T19:32:29.852 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54185.log 2024-05-17T19:32:29.852 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.36995.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36995.log.gz 2024-05-17T19:32:29.855 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61113.log 2024-05-17T19:32:29.855 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.54185.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54185.log.gz 2024-05-17T19:32:29.855 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58542.log 2024-05-17T19:32:29.856 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61113.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61113.log.gz 2024-05-17T19:32:29.856 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38574.log 2024-05-17T19:32:29.858 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.58542.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58542.log.gz 2024-05-17T19:32:29.858 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48455.log 2024-05-17T19:32:29.858 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.38574.log: 67.2% -- replaced with /var/log/ceph/cluster1-client.admin.38574.log.gz 2024-05-17T19:32:29.859 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56225.log 2024-05-17T19:32:29.861 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48455.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48455.log.gz 2024-05-17T19:32:29.861 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55437.log 2024-05-17T19:32:29.862 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56225.log: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56225.log.gz 2024-05-17T19:32:29.864 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56871.log 2024-05-17T19:32:29.864 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55437.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55437.log.gz 2024-05-17T19:32:29.864 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42599.log 2024-05-17T19:32:29.866 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.56871.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56871.log.gz 2024-05-17T19:32:29.866 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42857.log 2024-05-17T19:32:29.879 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.42599.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51635.log 2024-05-17T19:32:29.880 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.42857.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42857.log.gz 2024-05-17T19:32:29.885 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39909.log.gz 2024-05-17T19:32:29.887 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55863.log 2024-05-17T19:32:29.887 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51635.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51635.log.gz 2024-05-17T19:32:29.887 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55268.log 2024-05-17T19:32:29.887 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55863.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55863.log.gz 2024-05-17T19:32:29.889 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62982.log 2024-05-17T19:32:29.889 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55268.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55268.log.gz 2024-05-17T19:32:29.889 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61433.log 2024-05-17T19:32:29.891 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.62982.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62982.log.gz 2024-05-17T19:32:29.891 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44956.log 2024-05-17T19:32:29.891 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61433.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61433.log.gz 2024-05-17T19:32:29.892 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61173.log 2024-05-17T19:32:29.897 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.44956.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44956.log.gz 2024-05-17T19:32:29.898 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40878.log 2024-05-17T19:32:29.899 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61173.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61173.log.gz 2024-05-17T19:32:29.899 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52309.log 2024-05-17T19:32:29.905 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.40878.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40878.log.gz 2024-05-17T19:32:29.905 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61073.log 2024-05-17T19:32:29.907 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52309.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52309.log.gz 2024-05-17T19:32:29.907 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51615.log 2024-05-17T19:32:29.908 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61073.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61073.log.gz 2024-05-17T19:32:29.908 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44757.log 2024-05-17T19:32:29.929 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51615.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46447.log 2024-05-17T19:32:29.930 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.44757.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44757.log.gz 2024-05-17T19:32:29.943 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51536.log 2024-05-17T19:32:29.944 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46447.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46447.log.gz 2024-05-17T19:32:29.953 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51290.log.gz 2024-05-17T19:32:29.953 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47753.log 2024-05-17T19:32:29.976 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61533.log 2024-05-17T19:32:29.977 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47753.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47753.log.gz 2024-05-17T19:32:29.985 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55369.log 2024-05-17T19:32:29.985 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61533.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61533.log.gz 2024-05-17T19:32:29.997 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61193.log 2024-05-17T19:32:29.998 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.55369.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55369.log.gz 2024-05-17T19:32:30.011 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37321.log 2024-05-17T19:32:30.012 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61193.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61193.log.gz 2024-05-17T19:32:30.020 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52727.log 2024-05-17T19:32:30.021 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.37321.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37321.log.gz 2024-05-17T19:32:30.029 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59833.log 2024-05-17T19:32:30.030 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52727.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52727.log.gz 2024-05-17T19:32:30.038 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62611.log 2024-05-17T19:32:30.039 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59833.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59833.log.gz 2024-05-17T19:32:30.053 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62293.log 2024-05-17T19:32:30.054 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.62611.log: 51.6% -- replaced with /var/log/ceph/cluster1-client.admin.62611.log.gz 2024-05-17T19:32:30.062 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60793.log 2024-05-17T19:32:30.063 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62293.log: 87.4% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47854.log.gz 2024-05-17T19:32:30.063 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43344.log 2024-05-17T19:32:30.064 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60793.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60793.log.gz 2024-05-17T19:32:30.065 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52478.log 2024-05-17T19:32:30.065 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.43344.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43344.log.gz 2024-05-17T19:32:30.067 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58901.log 2024-05-17T19:32:30.067 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52478.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52478.log.gz 2024-05-17T19:32:30.068 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53326.log 2024-05-17T19:32:30.068 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.58901.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58901.log.gz 2024-05-17T19:32:30.069 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45416.log 2024-05-17T19:32:30.071 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53326.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.53326.log.gz 2024-05-17T19:32:30.072 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56022.log 2024-05-17T19:32:30.076 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45416.log: -- replaced with /var/log/ceph/cluster1-client.mirror.62293.log.gz 2024-05-17T19:32:30.077 INFO:teuthology.orchestra.run.smithi195.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45416.log.gz 2024-05-17T19:32:30.078 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62355.log 2024-05-17T19:32:30.079 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.56022.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56022.log.gz 2024-05-17T19:32:30.080 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.53913.log 2024-05-17T19:32:30.080 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62355.log: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62355.log.gz 2024-05-17T19:32:30.080 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42109.log 2024-05-17T19:32:30.089 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.0.53913.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.53913.log.gz 2024-05-17T19:32:30.089 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53438.log 2024-05-17T19:32:30.103 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.42109.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54426.log 2024-05-17T19:32:30.104 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53438.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53438.log.gz 2024-05-17T19:32:30.120 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62073.log 2024-05-17T19:32:30.130 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.54426.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54426.log.gz 2024-05-17T19:32:30.144 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37076.log 2024-05-17T19:32:30.145 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62073.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62073.log.gz 2024-05-17T19:32:30.153 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46487.log 2024-05-17T19:32:30.153 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.37076.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37076.log.gz 2024-05-17T19:32:30.171 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45107.log 2024-05-17T19:32:30.172 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46487.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46487.log.gz 2024-05-17T19:32:30.174 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47414.log.gz 2024-05-17T19:32:30.175 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63152.log 2024-05-17T19:32:30.175 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45107.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45107.log.gz 2024-05-17T19:32:30.177 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61373.log 2024-05-17T19:32:30.177 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63152.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63152.log.gz 2024-05-17T19:32:30.177 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52972.log 2024-05-17T19:32:30.177 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61373.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61373.log.gz 2024-05-17T19:32:30.179 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50518.log 2024-05-17T19:32:30.179 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52972.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52972.log.gz 2024-05-17T19:32:30.180 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49041.log 2024-05-17T19:32:30.180 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50518.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50518.log.gz 2024-05-17T19:32:30.180 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59473.log 2024-05-17T19:32:30.181 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49041.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49041.log.gz 2024-05-17T19:32:30.182 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62500.log 2024-05-17T19:32:30.182 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59473.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59473.log.gz 2024-05-17T19:32:30.182 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60613.log 2024-05-17T19:32:30.182 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.62500.log: 70.1% -- replaced with /var/log/ceph/cluster1-client.admin.62500.log.gz 2024-05-17T19:32:30.184 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63120.log 2024-05-17T19:32:30.184 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60613.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60613.log.gz 2024-05-17T19:32:30.184 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43848.log 2024-05-17T19:32:30.184 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63120.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63120.log.gz 2024-05-17T19:32:30.184 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55152.log 2024-05-17T19:32:30.185 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.43848.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43848.log.gz 2024-05-17T19:32:30.185 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48497.log 2024-05-17T19:32:30.186 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.55152.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55152.log.gz 2024-05-17T19:32:30.186 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44002.log 2024-05-17T19:32:30.196 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48497.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55309.log 2024-05-17T19:32:30.197 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.44002.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44002.log.gz 2024-05-17T19:32:30.210 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57514.log 2024-05-17T19:32:30.212 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55309.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55309.log.gz 2024-05-17T19:32:30.225 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44350.log 2024-05-17T19:32:30.227 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57514.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57514.log.gz 2024-05-17T19:32:30.240 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56112.log 2024-05-17T19:32:30.246 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.44350.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44350.log.gz 2024-05-17T19:32:30.259 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50599.log 2024-05-17T19:32:30.260 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56112.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56112.log.gz 2024-05-17T19:32:30.267 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48774.log 2024-05-17T19:32:30.267 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50599.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50599.log.gz 2024-05-17T19:32:30.275 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58365.log 2024-05-17T19:32:30.290 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57735.log 2024-05-17T19:32:30.291 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58365.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58365.log.gz 2024-05-17T19:32:30.291 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42599.log.gz 2024-05-17T19:32:30.312 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41039.log 2024-05-17T19:32:30.314 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57735.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57735.log.gz 2024-05-17T19:32:30.335 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51327.log 2024-05-17T19:32:30.335 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.41039.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.41039.log.gz 2024-05-17T19:32:30.347 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60093.log 2024-05-17T19:32:30.348 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51327.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51327.log.gz 2024-05-17T19:32:30.356 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52862.log 2024-05-17T19:32:30.362 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60093.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60093.log.gz 2024-05-17T19:32:30.369 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43505.log 2024-05-17T19:32:30.370 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.52862.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52862.log.gz 2024-05-17T19:32:30.378 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49162.log 2024-05-17T19:32:30.387 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.43505.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43505.log.gz 2024-05-17T19:32:30.392 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51536.log.gz 2024-05-17T19:32:30.393 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52213.log 2024-05-17T19:32:30.395 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49162.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49162.log.gz 2024-05-17T19:32:30.395 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52454.log 2024-05-17T19:32:30.396 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52213.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52213.log.gz 2024-05-17T19:32:30.396 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45201.log 2024-05-17T19:32:30.397 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52454.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52454.log.gz 2024-05-17T19:32:30.397 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37657.log 2024-05-17T19:32:30.404 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45201.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45201.log.gz 2024-05-17T19:32:30.404 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51031.log 2024-05-17T19:32:30.404 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.37657.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37657.log.gz 2024-05-17T19:32:30.406 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62577.log 2024-05-17T19:32:30.406 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51031.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51031.log.gz 2024-05-17T19:32:30.408 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44213.log 2024-05-17T19:32:30.408 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62577.log: 10.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62577.log.gz 2024-05-17T19:32:30.408 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45244.log 2024-05-17T19:32:30.409 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.44213.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44213.log.gz 2024-05-17T19:32:30.409 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40213.log 2024-05-17T19:32:30.415 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45244.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45244.log.gz 2024-05-17T19:32:30.416 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49063.log 2024-05-17T19:32:30.417 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.40213.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40213.log.gz 2024-05-17T19:32:30.417 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62153.log 2024-05-17T19:32:30.430 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49063.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61353.log 2024-05-17T19:32:30.431 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62153.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62153.log.gz 2024-05-17T19:32:30.448 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45823.log 2024-05-17T19:32:30.456 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61353.log: 87.2% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42109.log.gz 2024-05-17T19:32:30.456 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51615.log.gz 2024-05-17T19:32:30.456 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53575.log 2024-05-17T19:32:30.457 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62778.log 2024-05-17T19:32:30.460 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53575.log: /var/log/ceph/cluster1-client.mirror.45823.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53575.log.gz 2024-05-17T19:32:30.460 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41694.log 2024-05-17T19:32:30.460 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62778.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62778.log.gz 2024-05-17T19:32:30.460 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50555.log 2024-05-17T19:32:30.461 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.41694.log: -- replaced with /var/log/ceph/cluster1-client.mirror.61353.log.gz 2024-05-17T19:32:30.462 INFO:teuthology.orchestra.run.smithi195.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45823.log.gz 2024-05-17T19:32:30.463 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50794.log 2024-05-17T19:32:30.464 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50555.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50555.log.gz 2024-05-17T19:32:30.465 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56760.log 2024-05-17T19:32:30.466 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50794.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50794.log.gz 2024-05-17T19:32:30.466 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39069.log 2024-05-17T19:32:30.467 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56760.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56760.log.gz 2024-05-17T19:32:30.467 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37615.log 2024-05-17T19:32:30.468 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.39069.log: 55.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39069.log.gz 2024-05-17T19:32:30.468 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45287.log 2024-05-17T19:32:30.470 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.37615.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37615.log.gz 2024-05-17T19:32:30.470 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50150.log 2024-05-17T19:32:30.475 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45287.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45287.log.gz 2024-05-17T19:32:30.476 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45651.log 2024-05-17T19:32:30.476 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50150.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50150.log.gz 2024-05-17T19:32:30.477 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37447.log 2024-05-17T19:32:30.483 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45651.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45651.log.gz 2024-05-17T19:32:30.484 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37783.log 2024-05-17T19:32:30.485 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.37447.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37447.log.gz 2024-05-17T19:32:30.485 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59333.log 2024-05-17T19:32:30.486 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.37783.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37783.log.gz 2024-05-17T19:32:30.486 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59513.log 2024-05-17T19:32:30.487 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59333.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59333.log.gz 2024-05-17T19:32:30.487 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58010.log 2024-05-17T19:32:30.488 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59513.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59513.log.gz 2024-05-17T19:32:30.489 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59973.log 2024-05-17T19:32:30.490 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.58010.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58010.log.gz 2024-05-17T19:32:30.491 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41518.log 2024-05-17T19:32:30.492 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59973.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59973.log.gz 2024-05-17T19:32:30.492 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62445.log 2024-05-17T19:32:30.493 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.41518.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.41518.log.gz 2024-05-17T19:32:30.494 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45803.log 2024-05-17T19:32:30.494 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62445.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62445.log.gz 2024-05-17T19:32:30.495 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39086.log 2024-05-17T19:32:30.501 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45803.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45803.log.gz 2024-05-17T19:32:30.501 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47733.log 2024-05-17T19:32:30.502 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.39086.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39086.log.gz 2024-05-17T19:32:30.505 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38591.log 2024-05-17T19:32:30.508 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47733.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47733.log.gz 2024-05-17T19:32:30.509 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41066.log 2024-05-17T19:32:30.509 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.38591.log: 54.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38591.log.gz 2024-05-17T19:32:30.509 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56953.log 2024-05-17T19:32:30.509 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.41066.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.41066.log.gz 2024-05-17T19:32:30.509 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52002.log 2024-05-17T19:32:30.509 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56953.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56953.log.gz 2024-05-17T19:32:30.509 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58251.log 2024-05-17T19:32:30.521 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.52002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52840.log 2024-05-17T19:32:30.522 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.58251.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58251.log.gz 2024-05-17T19:32:30.536 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46996.log 2024-05-17T19:32:30.539 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52840.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52840.log.gz 2024-05-17T19:32:30.539 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40941.log 2024-05-17T19:32:30.540 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46996.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46996.log.gz 2024-05-17T19:32:30.549 INFO:teuthology.orchestra.run.smithi195.stderr: 93.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38659.log 2024-05-17T19:32:30.553 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.40941.log: -- replaced with /var/log/ceph/cluster2-client.mirror.52002.log.gz 2024-05-17T19:32:30.555 INFO:teuthology.orchestra.run.smithi195.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40941.log.gz 2024-05-17T19:32:30.555 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59296.log 2024-05-17T19:32:30.555 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.38659.log: 56.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38659.log.gz 2024-05-17T19:32:30.556 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48180.log 2024-05-17T19:32:30.556 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59296.log: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59296.log.gz 2024-05-17T19:32:30.557 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58413.log 2024-05-17T19:32:30.558 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48180.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48180.log.gz 2024-05-17T19:32:30.558 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60373.log 2024-05-17T19:32:30.559 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58413.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58413.log.gz 2024-05-17T19:32:30.559 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54122.log 2024-05-17T19:32:30.560 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60373.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60373.log.gz 2024-05-17T19:32:30.560 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43980.log 2024-05-17T19:32:30.561 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54122.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54122.log.gz 2024-05-17T19:32:30.562 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36867.log 2024-05-17T19:32:30.567 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.43980.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43980.log.gz 2024-05-17T19:32:30.568 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41607.log 2024-05-17T19:32:30.568 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.36867.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36867.log.gz 2024-05-17T19:32:30.569 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46610.log 2024-05-17T19:32:30.574 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.41607.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41607.log.gz 2024-05-17T19:32:30.575 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59154.log 2024-05-17T19:32:30.575 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.46610.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.46610.log.gz 2024-05-17T19:32:30.576 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59275.log 2024-05-17T19:32:30.576 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.59154.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59154.log.gz 2024-05-17T19:32:30.576 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52679.log 2024-05-17T19:32:30.589 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.59275.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.59275.log.gz 2024-05-17T19:32:30.590 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54698.log 2024-05-17T19:32:30.590 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.52679.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52679.log.gz 2024-05-17T19:32:30.591 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57875.log 2024-05-17T19:32:30.592 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54698.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54698.log.gz 2024-05-17T19:32:30.592 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63250.log 2024-05-17T19:32:30.593 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57875.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57875.log.gz 2024-05-17T19:32:30.593 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58140.log 2024-05-17T19:32:30.594 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63250.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63250.log.gz 2024-05-17T19:32:30.594 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51211.log 2024-05-17T19:32:30.595 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58140.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58140.log.gz 2024-05-17T19:32:30.595 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45479.log 2024-05-17T19:32:30.608 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51211.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41841.log 2024-05-17T19:32:30.610 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45479.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40418.log.gz 2024-05-17T19:32:30.614 INFO:teuthology.orchestra.run.smithi195.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45479.log.gz 2024-05-17T19:32:30.615 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56580.log 2024-05-17T19:32:30.616 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.41841.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41841.log.gz 2024-05-17T19:32:30.617 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53555.log 2024-05-17T19:32:30.619 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.56580.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.56580.log.gz 2024-05-17T19:32:30.619 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45588.log 2024-05-17T19:32:30.621 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.53555.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53555.log.gz 2024-05-17T19:32:30.622 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56136.log 2024-05-17T19:32:30.628 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45588.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45588.log.gz 2024-05-17T19:32:30.629 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60013.log 2024-05-17T19:32:30.630 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56136.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56136.log.gz 2024-05-17T19:32:30.630 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.38225.log 2024-05-17T19:32:30.631 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60013.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60013.log.gz 2024-05-17T19:32:30.632 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62113.log 2024-05-17T19:32:30.637 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.2.38225.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.2.38225.log.gz 2024-05-17T19:32:30.638 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46968.log 2024-05-17T19:32:30.639 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62113.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62113.log.gz 2024-05-17T19:32:30.639 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59573.log 2024-05-17T19:32:30.652 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59017.log 2024-05-17T19:32:30.653 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59573.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59573.log.gz 2024-05-17T19:32:30.666 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45565.log 2024-05-17T19:32:30.667 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59017.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59017.log.gz 2024-05-17T19:32:30.668 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.52002.log 2024-05-17T19:32:30.682 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45565.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45565.log.gz 2024-05-17T19:32:30.683 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50204.log 2024-05-17T19:32:30.696 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.0.52002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48349.log 2024-05-17T19:32:30.699 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.50204.log: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50204.log.gz 2024-05-17T19:32:30.712 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40333.log 2024-05-17T19:32:30.714 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48349.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48349.log.gz 2024-05-17T19:32:30.727 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52123.log 2024-05-17T19:32:30.728 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.40333.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40333.log.gz 2024-05-17T19:32:30.734 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37075.log 2024-05-17T19:32:30.735 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.52123.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52123.log.gz 2024-05-17T19:32:30.743 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59041.log 2024-05-17T19:32:30.743 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.37075.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37075.log.gz 2024-05-17T19:32:30.752 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58188.log 2024-05-17T19:32:30.753 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59041.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59041.log.gz 2024-05-17T19:32:30.761 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56487.log 2024-05-17T19:32:30.775 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59913.log 2024-05-17T19:32:30.775 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56487.log: 87.4% 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56487.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.58188.log.gz 2024-05-17T19:32:30.775 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-05-17T19:32:30.783 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49893.log 2024-05-17T19:32:30.784 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59913.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59913.log.gz 2024-05-17T19:32:30.799 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42535.log 2024-05-17T19:32:30.800 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49893.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49893.log.gz 2024-05-17T19:32:30.809 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48497.log.gz 2024-05-17T19:32:30.821 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51211.log.gz 2024-05-17T19:32:30.821 INFO:teuthology.orchestra.run.smithi195.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.1.48349.log 2024-05-17T19:32:30.821 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.42535.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38159.log 2024-05-17T19:32:30.826 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.1.48349.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42535.log.gz 2024-05-17T19:32:30.832 INFO:teuthology.orchestra.run.smithi195.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.48349.log.gz 2024-05-17T19:32:30.832 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37225.log 2024-05-17T19:32:30.835 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.38159.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.52002.log.gz 2024-05-17T19:32:30.836 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58341.log 2024-05-17T19:32:30.836 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.37225.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37225.log.gz 2024-05-17T19:32:30.837 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42471.log 2024-05-17T19:32:30.839 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58341.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58341.log.gz 2024-05-17T19:32:30.839 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38557.log 2024-05-17T19:32:30.840 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.42471.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42471.log.gz 2024-05-17T19:32:30.840 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60633.log 2024-05-17T19:32:30.841 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.38557.log: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.38557.log.gz 2024-05-17T19:32:30.841 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37867.log 2024-05-17T19:32:30.843 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60633.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60633.log.gz 2024-05-17T19:32:30.843 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60813.log 2024-05-17T19:32:30.843 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.37867.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37867.log.gz 2024-05-17T19:32:30.845 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47812.log 2024-05-17T19:32:30.845 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60813.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60813.log.gz 2024-05-17T19:32:30.845 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49406.log 2024-05-17T19:32:30.846 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47812.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47812.log.gz 2024-05-17T19:32:30.847 INFO:teuthology.orchestra.run.smithi195.stderr:gzip 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38159.log.gz 2024-05-17T19:32:30.847 INFO:teuthology.orchestra.run.smithi195.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48984.log 2024-05-17T19:32:30.848 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60773.log 2024-05-17T19:32:30.849 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49406.log: /var/log/ceph/cluster2-client.mirror.48984.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48774.log.gz 2024-05-17T19:32:30.851 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37699.log 2024-05-17T19:32:30.851 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60773.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60773.log.gz 2024-05-17T19:32:30.851 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48109.log 2024-05-17T19:32:30.852 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.37699.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37699.log.gz 2024-05-17T19:32:30.852 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48163.log 2024-05-17T19:32:30.853 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48109.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48109.log.gz 2024-05-17T19:32:30.853 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42087.log 2024-05-17T19:32:30.855 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48163.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48163.log.gz 2024-05-17T19:32:30.855 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44024.log 2024-05-17T19:32:30.856 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.42087.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42087.log.gz 2024-05-17T19:32:30.857 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41494.log 2024-05-17T19:32:30.866 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.44024.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45674.log 2024-05-17T19:32:30.867 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.41494.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41494.log.gz 2024-05-17T19:32:30.886 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55727.log 2024-05-17T19:32:30.904 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45674.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45674.log.gz 2024-05-17T19:32:30.908 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50122.log 2024-05-17T19:32:30.913 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.55727.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55727.log.gz 2024-05-17T19:32:30.923 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49617.log 2024-05-17T19:32:30.940 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50122.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54907.log 2024-05-17T19:32:30.942 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49617.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49617.log.gz 2024-05-17T19:32:30.951 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60713.log 2024-05-17T19:32:30.952 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.54907.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54907.log.gz 2024-05-17T19:32:30.960 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57715.log 2024-05-17T19:32:30.971 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60713.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43917.log 2024-05-17T19:32:30.973 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.57715.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57715.log.gz 2024-05-17T19:32:30.973 INFO:teuthology.orchestra.run.smithi195.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60713.log.gz 2024-05-17T19:32:30.980 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42423.log 2024-05-17T19:32:30.998 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.43917.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50915.log 2024-05-17T19:32:30.998 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.42423.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42423.log.gz 2024-05-17T19:32:30.998 INFO:teuthology.orchestra.run.smithi195.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43917.log.gz 2024-05-17T19:32:31.011 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41092.log 2024-05-17T19:32:31.011 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50915.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50915.log.gz 2024-05-17T19:32:31.028 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37573.log 2024-05-17T19:32:31.030 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.41092.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41092.log.gz 2024-05-17T19:32:31.034 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46968.log.gz 2024-05-17T19:32:31.047 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37909.log 2024-05-17T19:32:31.047 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.37573.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37573.log.gz 2024-05-17T19:32:31.051 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49063.log.gz 2024-05-17T19:32:31.053 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43416.log 2024-05-17T19:32:31.053 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.37909.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37909.log.gz 2024-05-17T19:32:31.053 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56511.log 2024-05-17T19:32:31.055 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.43416.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43416.log.gz 2024-05-17T19:32:31.056 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45760.log 2024-05-17T19:32:31.056 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56511.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56511.log.gz 2024-05-17T19:32:31.056 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61893.log 2024-05-17T19:32:31.061 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45760.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45760.log.gz 2024-05-17T19:32:31.063 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55959.log 2024-05-17T19:32:31.063 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61893.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61893.log.gz 2024-05-17T19:32:31.063 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45694.log 2024-05-17T19:32:31.064 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55959.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55959.log.gz 2024-05-17T19:32:31.064 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60493.log 2024-05-17T19:32:31.071 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45694.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45694.log.gz 2024-05-17T19:32:31.071 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55887.log 2024-05-17T19:32:31.072 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60493.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60493.log.gz 2024-05-17T19:32:31.072 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49735.log 2024-05-17T19:32:31.073 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55887.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55887.log.gz 2024-05-17T19:32:31.073 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50315.log 2024-05-17T19:32:31.075 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49735.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49735.log.gz 2024-05-17T19:32:31.075 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60873.log 2024-05-17T19:32:31.088 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50315.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38736.log 2024-05-17T19:32:31.089 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60873.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60873.log.gz 2024-05-17T19:32:31.100 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60973.log 2024-05-17T19:32:31.101 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.38736.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38736.log.gz 2024-05-17T19:32:31.114 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56849.log 2024-05-17T19:32:31.117 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60973.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60973.log.gz 2024-05-17T19:32:31.121 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48752.log 2024-05-17T19:32:31.123 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56849.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56849.log.gz 2024-05-17T19:32:31.130 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42898.log 2024-05-17T19:32:31.131 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48752.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48752.log.gz 2024-05-17T19:32:31.140 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39052.log 2024-05-17T19:32:31.149 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.42898.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49777.log 2024-05-17T19:32:31.149 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.39052.log: 67.4% -- replaced with /var/log/ceph/cluster2-client.admin.39052.log.gz 2024-05-17T19:32:31.158 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40374.log 2024-05-17T19:32:31.158 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49777.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42898.log.gz 2024-05-17T19:32:31.178 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53394.log 2024-05-17T19:32:31.182 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.40374.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40374.log.gz 2024-05-17T19:32:31.186 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57646.log 2024-05-17T19:32:31.187 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.53394.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53394.log.gz 2024-05-17T19:32:31.199 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48984.log.gz 2024-05-17T19:32:31.200 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46427.log 2024-05-17T19:32:31.201 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57646.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57646.log.gz 2024-05-17T19:32:31.215 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54074.log 2024-05-17T19:32:31.216 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46427.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46427.log.gz 2024-05-17T19:32:31.234 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55548.log 2024-05-17T19:32:31.235 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54074.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54074.log.gz 2024-05-17T19:32:31.248 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50715.log 2024-05-17T19:32:31.249 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.55548.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55548.log.gz 2024-05-17T19:32:31.259 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61753.log 2024-05-17T19:32:31.259 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50715.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50715.log.gz 2024-05-17T19:32:31.267 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61853.log 2024-05-17T19:32:31.268 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61753.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61753.log.gz 2024-05-17T19:32:31.276 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49580.log 2024-05-17T19:32:31.277 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61853.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61853.log.gz 2024-05-17T19:32:31.293 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61913.log 2024-05-17T19:32:31.302 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49580.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54582.log 2024-05-17T19:32:31.303 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61913.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61913.log.gz 2024-05-17T19:32:31.311 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50873.log 2024-05-17T19:32:31.312 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.54582.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54582.log.gz 2024-05-17T19:32:31.322 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43592.log 2024-05-17T19:32:31.324 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50873.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50873.log.gz 2024-05-17T19:32:31.336 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45866.log 2024-05-17T19:32:31.353 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.43592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53727.log 2024-05-17T19:32:31.359 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45866.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45866.log.gz 2024-05-17T19:32:31.362 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59853.log 2024-05-17T19:32:31.363 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53727.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53727.log.gz 2024-05-17T19:32:31.366 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49406.log.gz 2024-05-17T19:32:31.375 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40761.log 2024-05-17T19:32:31.376 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59853.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59853.log.gz 2024-05-17T19:32:31.389 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60513.log 2024-05-17T19:32:31.390 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.40761.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40761.log.gz 2024-05-17T19:32:31.402 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49876.log 2024-05-17T19:32:31.403 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60513.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60513.log.gz 2024-05-17T19:32:31.415 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59313.log 2024-05-17T19:32:31.416 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49876.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49876.log.gz 2024-05-17T19:32:31.423 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43937.log 2024-05-17T19:32:31.424 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59313.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59313.log.gz 2024-05-17T19:32:31.425 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41694.log.gz 2024-05-17T19:32:31.425 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.41129.log 2024-05-17T19:32:31.430 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.43937.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43937.log.gz 2024-05-17T19:32:31.436 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40921.log 2024-05-17T19:32:31.440 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.0.41129.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49777.log.gz 2024-05-17T19:32:31.448 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50952.log 2024-05-17T19:32:31.458 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.40921.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62629.log 2024-05-17T19:32:31.458 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40921.log.gz 2024-05-17T19:32:31.459 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50952.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50952.log.gz 2024-05-17T19:32:31.468 INFO:teuthology.orchestra.run.smithi195.stderr: 92.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47355.log 2024-05-17T19:32:31.468 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62629.log: 11.2% 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62629.log.gz 2024-05-17T19:32:31.468 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.50315.log.gz 2024-05-17T19:32:31.469 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.0.41129.log.gz 2024-05-17T19:32:31.469 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41981.log 2024-05-17T19:32:31.472 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47355.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55911.log 2024-05-17T19:32:31.472 INFO:teuthology.orchestra.run.smithi195.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47355.log.gz 2024-05-17T19:32:31.472 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.41981.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41981.log.gz 2024-05-17T19:32:31.472 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51477.log 2024-05-17T19:32:31.472 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.55911.log: --verbose -- /var/log/ceph/cluster2-client.mirror.1.40610.log 2024-05-17T19:32:31.472 INFO:teuthology.orchestra.run.smithi195.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55911.log.gz 2024-05-17T19:32:31.472 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51477.log: 80.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57923.log 2024-05-17T19:32:31.472 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.1.40610.log: -- replaced with /var/log/ceph/cluster2-client.mirror.51477.log.gz 2024-05-17T19:32:31.473 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61233.log 2024-05-17T19:32:31.473 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57923.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57923.log.gz 2024-05-17T19:32:31.473 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.47534.log 2024-05-17T19:32:31.475 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61233.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.61233.log.gz 2024-05-17T19:32:31.475 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57536.log 2024-05-17T19:32:31.478 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.3.47534.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.1.40610.log.gz 2024-05-17T19:32:31.479 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.44574.log 2024-05-17T19:32:31.480 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.57536.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.3.47534.log.gz 2024-05-17T19:32:31.480 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46695.log 2024-05-17T19:32:31.480 INFO:teuthology.orchestra.run.smithi195.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57536.log.gz 2024-05-17T19:32:31.480 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.1.44574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60733.log 2024-05-17T19:32:31.480 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46695.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46695.log.gz 2024-05-17T19:32:31.481 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55610.log 2024-05-17T19:32:31.482 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60733.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60733.log.gz 2024-05-17T19:32:31.483 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45373.log 2024-05-17T19:32:31.484 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.55610.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55610.log.gz 2024-05-17T19:32:31.484 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56600.log 2024-05-17T19:32:31.490 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45373.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45373.log.gz 2024-05-17T19:32:31.491 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55769.log 2024-05-17T19:32:31.492 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56600.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56600.log.gz 2024-05-17T19:32:31.493 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61733.log 2024-05-17T19:32:31.495 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.55769.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55769.log.gz 2024-05-17T19:32:31.495 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41426.log 2024-05-17T19:32:31.496 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61733.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61733.log.gz 2024-05-17T19:32:31.496 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37825.log 2024-05-17T19:32:31.498 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.41426.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41426.log.gz 2024-05-17T19:32:31.499 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48547.log 2024-05-17T19:32:31.499 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.37825.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37825.log.gz 2024-05-17T19:32:31.500 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60453.log 2024-05-17T19:32:31.501 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.48547.log: 94.6% -- replaced with /var/log/ceph/cluster1-client.mirror.48547.log.gz 2024-05-17T19:32:31.501 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49797.log 2024-05-17T19:32:31.503 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60453.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60453.log.gz 2024-05-17T19:32:31.503 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56668.log 2024-05-17T19:32:31.504 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49797.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49797.log.gz 2024-05-17T19:32:31.504 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53285.log 2024-05-17T19:32:31.505 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.56668.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56668.log.gz 2024-05-17T19:32:31.506 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42647.log 2024-05-17T19:32:31.507 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53285.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53285.log.gz 2024-05-17T19:32:31.507 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48200.log 2024-05-17T19:32:31.508 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.42647.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.42647.log.gz 2024-05-17T19:32:31.509 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44330.log 2024-05-17T19:32:31.509 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48200.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48200.log.gz 2024-05-17T19:32:31.510 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50577.log 2024-05-17T19:32:31.516 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.44330.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44330.log.gz 2024-05-17T19:32:31.517 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55220.log 2024-05-17T19:32:31.530 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50577.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37531.log 2024-05-17T19:32:31.531 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55220.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55220.log.gz 2024-05-17T19:32:31.539 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53085.log 2024-05-17T19:32:31.539 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.37531.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37531.log.gz 2024-05-17T19:32:31.540 INFO:teuthology.orchestra.run.smithi195.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.1.44574.log.gz 2024-05-17T19:32:31.540 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59813.log 2024-05-17T19:32:31.543 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53085.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53085.log.gz 2024-05-17T19:32:31.543 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50695.log 2024-05-17T19:32:31.544 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59813.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59813.log.gz 2024-05-17T19:32:31.544 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44709.log 2024-05-17T19:32:31.546 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50695.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50695.log.gz 2024-05-17T19:32:31.546 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49100.log 2024-05-17T19:32:31.548 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.44709.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44709.log.gz 2024-05-17T19:32:31.548 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49558.log 2024-05-17T19:32:31.549 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49100.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49100.log.gz 2024-05-17T19:32:31.549 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48089.log 2024-05-17T19:32:31.551 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49558.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49558.log.gz 2024-05-17T19:32:31.551 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61873.log 2024-05-17T19:32:31.553 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48089.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48089.log.gz 2024-05-17T19:32:31.553 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43800.log 2024-05-17T19:32:31.553 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61873.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61873.log.gz 2024-05-17T19:32:31.554 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33615.log 2024-05-17T19:32:31.556 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.43800.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43800.log.gz 2024-05-17T19:32:31.556 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58500.log 2024-05-17T19:32:31.556 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.33615.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33615.log.gz 2024-05-17T19:32:31.557 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61973.log 2024-05-17T19:32:31.558 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.58500.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58500.log.gz 2024-05-17T19:32:31.559 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45330.log 2024-05-17T19:32:31.560 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61973.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61973.log.gz 2024-05-17T19:32:31.560 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51672.log 2024-05-17T19:32:31.566 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45330.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45330.log.gz 2024-05-17T19:32:31.567 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60833.log 2024-05-17T19:32:31.568 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51672.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51672.log.gz 2024-05-17T19:32:31.568 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.38009.log 2024-05-17T19:32:31.568 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60833.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60833.log.gz 2024-05-17T19:32:31.569 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50535.log 2024-05-17T19:32:31.583 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.1.38009.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54602.log 2024-05-17T19:32:31.585 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50535.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50535.log.gz 2024-05-17T19:32:31.588 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50122.log.gz 2024-05-17T19:32:31.589 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43073.log 2024-05-17T19:32:31.590 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.54602.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54602.log.gz 2024-05-17T19:32:31.591 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45436.log 2024-05-17T19:32:31.592 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.43073.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43073.log.gz 2024-05-17T19:32:31.593 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58092.log 2024-05-17T19:32:31.599 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45436.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45436.log.gz 2024-05-17T19:32:31.599 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56622.log 2024-05-17T19:32:31.600 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58092.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58092.log.gz 2024-05-17T19:32:31.601 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59673.log 2024-05-17T19:32:31.603 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.56622.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56622.log.gz 2024-05-17T19:32:31.603 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44822.log 2024-05-17T19:32:31.604 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59673.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59673.log.gz 2024-05-17T19:32:31.604 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57046.log 2024-05-17T19:32:31.606 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.44822.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44822.log.gz 2024-05-17T19:32:31.606 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58634.log 2024-05-17T19:32:31.608 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.57046.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57046.log.gz 2024-05-17T19:32:31.609 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53462.log 2024-05-17T19:32:31.610 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58634.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58634.log.gz 2024-05-17T19:32:31.610 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59533.log 2024-05-17T19:32:31.611 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53462.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53462.log.gz 2024-05-17T19:32:31.612 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60113.log 2024-05-17T19:32:31.613 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59533.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59533.log.gz 2024-05-17T19:32:31.613 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60413.log 2024-05-17T19:32:31.614 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60113.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60113.log.gz 2024-05-17T19:32:31.615 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54674.log 2024-05-17T19:32:31.616 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60413.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60413.log.gz 2024-05-17T19:32:31.616 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45350.log 2024-05-17T19:32:31.617 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54674.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54674.log.gz 2024-05-17T19:32:31.617 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52330.log 2024-05-17T19:32:31.623 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45350.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45350.log.gz 2024-05-17T19:32:31.624 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54339.log 2024-05-17T19:32:31.626 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.52330.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52330.log.gz 2024-05-17T19:32:31.626 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56559.log 2024-05-17T19:32:31.627 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54339.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54339.log.gz 2024-05-17T19:32:31.627 INFO:teuthology.orchestra.run.smithi195.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.38009.log.gz 2024-05-17T19:32:31.628 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38693.log 2024-05-17T19:32:31.628 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56559.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56559.log.gz 2024-05-17T19:32:31.629 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54975.log 2024-05-17T19:32:31.630 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.38693.log: --verbose -- /var/log/ceph/cluster2-client.mirror.40396.log 2024-05-17T19:32:31.630 INFO:teuthology.orchestra.run.smithi195.stderr: 54.5% -- replaced with /var/log/ceph/cluster1-client.mirror.38693.log.gz 2024-05-17T19:32:31.630 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60473.log 2024-05-17T19:32:31.632 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54975.log: /var/log/ceph/cluster2-client.mirror.40396.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54975.log.gz 2024-05-17T19:32:31.632 INFO:teuthology.orchestra.run.smithi195.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40396.log.gz 2024-05-17T19:32:31.632 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44169.log 2024-05-17T19:32:31.632 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60473.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48846.log 2024-05-17T19:32:31.632 INFO:teuthology.orchestra.run.smithi195.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60473.log.gz 2024-05-17T19:32:31.634 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.44169.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52571.log 2024-05-17T19:32:31.634 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48846.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48846.log.gz 2024-05-17T19:32:31.634 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50377.log 2024-05-17T19:32:31.635 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.52571.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44169.log.gz 2024-05-17T19:32:31.635 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56736.log 2024-05-17T19:32:31.636 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50377.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50377.log.gz 2024-05-17T19:32:31.636 INFO:teuthology.orchestra.run.smithi195.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52571.log.gz 2024-05-17T19:32:31.636 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40857.log 2024-05-17T19:32:31.636 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58164.log 2024-05-17T19:32:31.638 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56736.log: /var/log/ceph/cluster1-client.mirror.40857.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56736.log.gz 2024-05-17T19:32:31.638 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37951.log 2024-05-17T19:32:31.639 INFO:teuthology.orchestra.run.smithi195.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40857.log.gz 2024-05-17T19:32:31.639 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58164.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48673.log 2024-05-17T19:32:31.639 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.37951.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37951.log.gz 2024-05-17T19:32:31.639 INFO:teuthology.orchestra.run.smithi195.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58164.log.gz 2024-05-17T19:32:31.639 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48222.log 2024-05-17T19:32:31.639 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61253.log 2024-05-17T19:32:31.639 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48673.log: /var/log/ceph/cluster2-client.mirror.48222.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48673.log.gz 2024-05-17T19:32:31.640 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62757.log 2024-05-17T19:32:31.640 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61253.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61253.log.gz 2024-05-17T19:32:31.640 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49600.log 2024-05-17T19:32:31.641 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62757.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62757.log.gz 2024-05-17T19:32:31.641 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47455.log 2024-05-17T19:32:31.641 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49600.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49600.log.gz 2024-05-17T19:32:31.642 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62936.log 2024-05-17T19:32:31.643 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47455.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47455.log.gz 2024-05-17T19:32:31.643 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46777.log 2024-05-17T19:32:31.644 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.62936.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.62936.log.gz 2024-05-17T19:32:31.644 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41799.log 2024-05-17T19:32:31.645 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46777.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46777.log.gz 2024-05-17T19:32:31.645 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52996.log 2024-05-17T19:32:31.647 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.41799.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41799.log.gz 2024-05-17T19:32:31.647 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47285.log 2024-05-17T19:32:31.648 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52996.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52996.log.gz 2024-05-17T19:32:31.648 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53679.log 2024-05-17T19:32:31.649 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47285.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47285.log.gz 2024-05-17T19:32:31.650 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38625.log 2024-05-17T19:32:31.651 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53679.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53679.log.gz 2024-05-17T19:32:31.651 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46018.log 2024-05-17T19:32:31.651 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.38625.log: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.38625.log.gz 2024-05-17T19:32:31.652 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38009.log 2024-05-17T19:32:31.656 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46018.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44024.log.gz 2024-05-17T19:32:31.658 INFO:teuthology.orchestra.run.smithi195.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46018.log.gz 2024-05-17T19:32:31.658 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59353.log 2024-05-17T19:32:31.659 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46307.log 2024-05-17T19:32:31.659 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.38009.log: /var/log/ceph/cluster1-client.mirror.59353.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59353.log.gz 2024-05-17T19:32:31.669 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48715.log 2024-05-17T19:32:31.672 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46307.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46307.log.gz 2024-05-17T19:32:31.672 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43872.log 2024-05-17T19:32:31.672 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48715.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48715.log.gz 2024-05-17T19:32:31.672 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61933.log 2024-05-17T19:32:31.673 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.43872.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43872.log.gz 2024-05-17T19:32:31.673 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53703.log 2024-05-17T19:32:31.674 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61933.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61933.log.gz 2024-05-17T19:32:31.674 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38225.log 2024-05-17T19:32:31.675 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53703.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53703.log.gz 2024-05-17T19:32:31.675 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62889.log 2024-05-17T19:32:31.676 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.38225.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.38225.log.gz 2024-05-17T19:32:31.677 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.38002.log 2024-05-17T19:32:31.678 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62889.log: 27.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62889.log.gz 2024-05-17T19:32:31.678 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44484.log 2024-05-17T19:32:31.694 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.0.38002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47179.log 2024-05-17T19:32:31.695 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.44484.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.44484.log.gz 2024-05-17T19:32:31.711 INFO:teuthology.orchestra.run.smithi195.stderr: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.38002.log.gz 2024-05-17T19:32:31.711 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51732.log 2024-05-17T19:32:31.711 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47179.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46081.log 2024-05-17T19:32:31.713 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.51732.log: 95.1% -- replaced with /var/log/ceph/cluster1-client.mirror.51732.log.gz 2024-05-17T19:32:31.713 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54026.log 2024-05-17T19:32:31.714 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46081.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46081.log.gz 2024-05-17T19:32:31.715 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56463.log 2024-05-17T19:32:31.716 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54026.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54026.log.gz 2024-05-17T19:32:31.717 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62970.log 2024-05-17T19:32:31.718 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.56463.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56463.log.gz 2024-05-17T19:32:31.718 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49258.log 2024-05-17T19:32:31.719 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.62970.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.62970.log.gz 2024-05-17T19:32:31.719 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60953.log 2024-05-17T19:32:31.733 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49258.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49258.log.gz 2024-05-17T19:32:31.733 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41909.log 2024-05-17T19:32:31.733 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60953.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60953.log.gz 2024-05-17T19:32:31.734 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62193.log 2024-05-17T19:32:31.734 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.41909.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41909.log.gz 2024-05-17T19:32:31.734 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58116.log 2024-05-17T19:32:31.734 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62193.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62193.log.gz 2024-05-17T19:32:31.734 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45932.log 2024-05-17T19:32:31.734 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.58116.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58116.log.gz 2024-05-17T19:32:31.734 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39035.log 2024-05-17T19:32:31.738 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.45932.log: 94.3% 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45932.log.gz 2024-05-17T19:32:31.739 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48045.log 2024-05-17T19:32:31.739 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.39035.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.39035.log.gz 2024-05-17T19:32:31.740 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57357.log 2024-05-17T19:32:31.753 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48045.log: -- replaced with /var/log/ceph/cluster2-client.mirror.38009.log.gz 2024-05-17T19:32:31.753 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51248.log 2024-05-17T19:32:31.754 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.57357.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57357.log.gz 2024-05-17T19:32:31.755 INFO:teuthology.orchestra.run.smithi195.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47179.log.gz 2024-05-17T19:32:31.756 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54650.log 2024-05-17T19:32:31.757 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51248.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51248.log.gz 2024-05-17T19:32:31.759 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62740.log 2024-05-17T19:32:31.759 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54650.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54650.log.gz 2024-05-17T19:32:31.760 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46740.log 2024-05-17T19:32:31.760 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62740.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62740.log.gz 2024-05-17T19:32:31.761 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38071.log 2024-05-17T19:32:31.762 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46740.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46740.log.gz 2024-05-17T19:32:31.762 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61613.log 2024-05-17T19:32:31.775 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.38071.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62213.log 2024-05-17T19:32:31.776 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61613.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61613.log.gz 2024-05-17T19:32:31.789 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62948.log 2024-05-17T19:32:31.790 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62213.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62213.log.gz 2024-05-17T19:32:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47392.log 2024-05-17T19:32:31.791 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.62948.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62948.log.gz 2024-05-17T19:32:31.804 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57578.log 2024-05-17T19:32:31.805 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47392.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47392.log.gz 2024-05-17T19:32:31.811 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59413.log 2024-05-17T19:32:31.816 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.57578.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57578.log.gz 2024-05-17T19:32:31.824 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55172.log 2024-05-17T19:32:31.824 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59413.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59413.log.gz 2024-05-17T19:32:31.828 INFO:teuthology.orchestra.run.smithi195.stderr: 94.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38071.log.gz 2024-05-17T19:32:31.829 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42921.log 2024-05-17T19:32:31.830 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.55172.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55172.log.gz 2024-05-17T19:32:31.831 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46387.log 2024-05-17T19:32:31.836 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.42921.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42921.log.gz 2024-05-17T19:32:31.837 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49755.log 2024-05-17T19:32:31.838 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46387.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46387.log.gz 2024-05-17T19:32:31.839 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33241.log 2024-05-17T19:32:31.839 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49755.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49755.log.gz 2024-05-17T19:32:31.840 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62424.log 2024-05-17T19:32:31.841 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.33241.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33241.log.gz 2024-05-17T19:32:31.842 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61713.log 2024-05-17T19:32:31.842 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62424.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62424.log.gz 2024-05-17T19:32:31.843 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-05-17T19:32:31.844 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61713.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61713.log.gz 2024-05-17T19:32:31.845 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48290.log 2024-05-17T19:32:31.845 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43592.log.gz 2024-05-17T19:32:31.845 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54951.log 2024-05-17T19:32:31.847 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48290.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48290.log.gz 2024-05-17T19:32:31.847 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40237.log 2024-05-17T19:32:31.847 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54951.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54951.log.gz 2024-05-17T19:32:31.847 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49718.log 2024-05-17T19:32:31.848 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.40237.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40237.log.gz 2024-05-17T19:32:31.848 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42785.log 2024-05-17T19:32:31.848 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49718.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49718.log.gz 2024-05-17T19:32:31.850 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58458.log 2024-05-17T19:32:31.850 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.42785.log: 88.0% -- replaced with /var/log/ceph/cluster2.log.gz 2024-05-17T19:32:31.850 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53838.log 2024-05-17T19:32:31.850 INFO:teuthology.orchestra.run.smithi195.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42785.log.gz 2024-05-17T19:32:31.850 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.58458.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47142.log 2024-05-17T19:32:31.851 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.53838.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53838.log.gz 2024-05-17T19:32:31.852 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43464.log 2024-05-17T19:32:31.852 INFO:teuthology.orchestra.run.smithi195.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58458.log.gz 2024-05-17T19:32:31.852 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51073.log 2024-05-17T19:32:31.854 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.43464.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43464.log.gz 2024-05-17T19:32:31.854 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58921.log 2024-05-17T19:32:31.854 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51073.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51073.log.gz 2024-05-17T19:32:31.855 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46588.log 2024-05-17T19:32:31.856 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.58921.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58921.log.gz 2024-05-17T19:32:31.856 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61333.log 2024-05-17T19:32:31.859 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46588.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46588.log.gz 2024-05-17T19:32:31.859 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57377.log 2024-05-17T19:32:31.859 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61333.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61333.log.gz 2024-05-17T19:32:31.861 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40193.log 2024-05-17T19:32:31.861 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.57377.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57377.log.gz 2024-05-17T19:32:31.861 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48695.log 2024-05-17T19:32:31.862 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.40193.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40193.log.gz 2024-05-17T19:32:31.863 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39193.log 2024-05-17T19:32:31.875 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48695.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38248.log 2024-05-17T19:32:31.876 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.39193.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39193.log.gz 2024-05-17T19:32:31.882 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41542.log 2024-05-17T19:32:31.895 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.38248.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.38248.log.gz 2024-05-17T19:32:31.898 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55706.log 2024-05-17T19:32:31.899 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.41542.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41542.log.gz 2024-05-17T19:32:31.905 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41587.log 2024-05-17T19:32:31.906 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55706.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55706.log.gz 2024-05-17T19:32:31.920 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59104.log 2024-05-17T19:32:31.926 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.41587.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41587.log.gz 2024-05-17T19:32:31.930 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47100.log 2024-05-17T19:32:31.931 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.59104.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59104.log.gz 2024-05-17T19:32:31.942 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44999.log 2024-05-17T19:32:31.943 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47100.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47100.log.gz 2024-05-17T19:32:31.949 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57494.log 2024-05-17T19:32:31.959 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.44999.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56419.log 2024-05-17T19:32:31.967 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.57494.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44999.log.gz 2024-05-17T19:32:31.971 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44373.log 2024-05-17T19:32:31.972 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.56419.log: 89.6% 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56419.log.gz 2024-05-17T19:32:31.972 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.57494.log.gz 2024-05-17T19:32:31.987 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55634.log 2024-05-17T19:32:32.000 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.44373.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44373.log.gz 2024-05-17T19:32:32.002 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49580.log.gz 2024-05-17T19:32:32.003 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61813.log 2024-05-17T19:32:32.003 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55634.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55634.log.gz 2024-05-17T19:32:32.004 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51310.log 2024-05-17T19:32:32.004 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61813.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61813.log.gz 2024-05-17T19:32:32.007 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50360.log 2024-05-17T19:32:32.007 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51310.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51310.log.gz 2024-05-17T19:32:32.008 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37489.log 2024-05-17T19:32:32.008 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50360.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50360.log.gz 2024-05-17T19:32:32.008 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51415.log 2024-05-17T19:32:32.008 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.37489.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37489.log.gz 2024-05-17T19:32:32.008 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55068.log 2024-05-17T19:32:32.009 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51415.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51415.log.gz 2024-05-17T19:32:32.009 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61413.log 2024-05-17T19:32:32.010 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.55068.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55068.log.gz 2024-05-17T19:32:32.011 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-05-17T19:32:32.011 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61413.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.61413.log.gz 2024-05-17T19:32:32.012 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55331.log 2024-05-17T19:32:32.015 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2.audit.log: 92.6% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-05-17T19:32:32.015 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54243.log 2024-05-17T19:32:32.016 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.55331.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55331.log.gz 2024-05-17T19:32:32.016 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50063.log 2024-05-17T19:32:32.017 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.54243.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54243.log.gz 2024-05-17T19:32:32.017 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54291.log 2024-05-17T19:32:32.018 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50063.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50063.log.gz 2024-05-17T19:32:32.019 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53044.log 2024-05-17T19:32:32.019 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54291.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54291.log.gz 2024-05-17T19:32:32.019 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50816.log 2024-05-17T19:32:32.020 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53044.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53044.log.gz 2024-05-17T19:32:32.021 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43047.log 2024-05-17T19:32:32.036 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50816.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38676.log 2024-05-17T19:32:32.037 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.43047.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.43047.log.gz 2024-05-17T19:32:32.038 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48222.log.gz 2024-05-17T19:32:32.038 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54999.log 2024-05-17T19:32:32.039 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.38676.log: 53.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38676.log.gz 2024-05-17T19:32:32.040 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53020.log 2024-05-17T19:32:32.041 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54999.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54999.log.gz 2024-05-17T19:32:32.041 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51593.log 2024-05-17T19:32:32.042 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53020.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53020.log.gz 2024-05-17T19:32:32.043 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59062.log 2024-05-17T19:32:32.043 INFO:teuthology.orchestra.run.smithi195.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48045.log.gz 2024-05-17T19:32:32.043 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51593.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51593.log.gz 2024-05-17T19:32:32.044 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59082.log 2024-05-17T19:32:32.044 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44070.log 2024-05-17T19:32:32.046 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.59062.log: /var/log/ceph/cluster1-client.mirror.59082.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59062.log.gz 2024-05-17T19:32:32.046 INFO:teuthology.orchestra.run.smithi195.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59082.log.gz 2024-05-17T19:32:32.047 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.54481.log 2024-05-17T19:32:32.047 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.44070.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.44070.log.gz 2024-05-17T19:32:32.047 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42673.log 2024-05-17T19:32:32.048 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.49973.log 2024-05-17T19:32:32.049 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.1.54481.log: /var/log/ceph/cluster2-client.mirror.42673.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42673.log.gz 2024-05-17T19:32:32.056 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50439.log 2024-05-17T19:32:32.074 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.0.49973.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40261.log 2024-05-17T19:32:32.076 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60933.log 2024-05-17T19:32:32.076 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.50439.log: --verbose -- /var/log/ceph/cluster1-client.mirror.62688.log 2024-05-17T19:32:32.078 INFO:teuthology.orchestra.run.smithi195.stderr:gzip/var/log/ceph/cluster1-client.mirror.40261.log: 87.7% -5 --verbose -- 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50439.log.gz 2024-05-17T19:32:32.078 INFO:teuthology.orchestra.run.smithi195.stderr: /var/log/ceph/cluster2-client.mirror.56929.log 2024-05-17T19:32:32.078 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.40261.log.gz 2024-05-17T19:32:32.079 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60933.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60933.log.gz 2024-05-17T19:32:32.080 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62688.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62688.log.gz 2024-05-17T19:32:32.080 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60993.log 2024-05-17T19:32:32.081 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.56929.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56929.log.gz 2024-05-17T19:32:32.081 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55658.log 2024-05-17T19:32:32.082 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60993.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60993.log.gz 2024-05-17T19:32:32.082 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47033.log 2024-05-17T19:32:32.083 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.55658.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55658.log.gz 2024-05-17T19:32:32.084 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62483.log 2024-05-17T19:32:32.086 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47033.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47033.log.gz 2024-05-17T19:32:32.086 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49856.log 2024-05-17T19:32:32.086 INFO:teuthology.orchestra.run.smithi195.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.0.49973.log.gz 2024-05-17T19:32:32.086 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62483.log: 48.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62483.log.gz 2024-05-17T19:32:32.086 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60673.log 2024-05-17T19:32:32.102 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49856.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50577.log.gz 2024-05-17T19:32:32.103 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60393.log 2024-05-17T19:32:32.104 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60673.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60673.log.gz 2024-05-17T19:32:32.105 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63239.log 2024-05-17T19:32:32.105 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60393.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60393.log.gz 2024-05-17T19:32:32.108 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61473.log 2024-05-17T19:32:32.108 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.63239.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.admin.63239.log.gz 2024-05-17T19:32:32.108 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61313.log 2024-05-17T19:32:32.108 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61473.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61473.log.gz 2024-05-17T19:32:32.108 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62667.log 2024-05-17T19:32:32.109 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61313.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61313.log.gz 2024-05-17T19:32:32.109 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61133.log 2024-05-17T19:32:32.109 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62667.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62667.log.gz 2024-05-17T19:32:32.110 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39230.log 2024-05-17T19:32:32.111 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61133.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61133.log.gz 2024-05-17T19:32:32.111 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59214.log 2024-05-17T19:32:32.111 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.39230.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39230.log.gz 2024-05-17T19:32:32.112 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44733.log 2024-05-17T19:32:32.113 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.59214.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.59214.log.gz 2024-05-17T19:32:32.113 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50895.log 2024-05-17T19:32:32.115 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.44733.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44733.log.gz 2024-05-17T19:32:32.115 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52189.log 2024-05-17T19:32:32.137 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40102.log 2024-05-17T19:32:32.137 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.52189.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52189.log.gz 2024-05-17T19:32:32.150 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42577.log 2024-05-17T19:32:32.151 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.40102.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.40102.log.gz 2024-05-17T19:32:32.157 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47327.log 2024-05-17T19:32:32.158 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.42577.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42577.log.gz 2024-05-17T19:32:32.166 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41129.log 2024-05-17T19:32:32.175 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.47327.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54315.log 2024-05-17T19:32:32.178 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.41129.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41129.log.gz 2024-05-17T19:32:32.187 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33553.log 2024-05-17T19:32:32.187 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54315.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54315.log.gz 2024-05-17T19:32:32.203 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62334.log 2024-05-17T19:32:32.203 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.33553.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33553.log.gz 2024-05-17T19:32:32.212 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54050.log 2024-05-17T19:32:32.213 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62334.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62334.log.gz 2024-05-17T19:32:32.230 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57899.log 2024-05-17T19:32:32.230 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.54050.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54050.log.gz 2024-05-17T19:32:32.247 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49973.log 2024-05-17T19:32:32.248 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.57899.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57899.log.gz 2024-05-17T19:32:32.260 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48732.log 2024-05-17T19:32:32.274 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49973.log: 92.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49199.log 2024-05-17T19:32:32.275 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48732.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48732.log.gz 2024-05-17T19:32:32.277 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49973.log.gz 2024-05-17T19:32:32.284 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60073.log 2024-05-17T19:32:32.285 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49199.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49199.log.gz 2024-05-17T19:32:32.300 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46061.log 2024-05-17T19:32:32.300 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60073.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60073.log.gz 2024-05-17T19:32:32.308 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57598.log 2024-05-17T19:32:32.321 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46061.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46061.log.gz 2024-05-17T19:32:32.327 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55389.log 2024-05-17T19:32:32.328 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.57598.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57598.log.gz 2024-05-17T19:32:32.346 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.38171.log 2024-05-17T19:32:32.347 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.55389.log: 87.4% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50895.log.gz 2024-05-17T19:32:32.347 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55389.log.gz 2024-05-17T19:32:32.360 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41746.log 2024-05-17T19:32:32.370 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.0.38171.log: 92.1%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62372.log 2024-05-17T19:32:32.371 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.41746.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.41746.log.gz 2024-05-17T19:32:32.383 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.0.38171.log.gz 2024-05-17T19:32:32.392 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47142.log.gz 2024-05-17T19:32:32.392 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59793.log 2024-05-17T19:32:32.393 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62372.log: 49.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62372.log.gz 2024-05-17T19:32:32.393 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45780.log 2024-05-17T19:32:32.394 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59793.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59793.log.gz 2024-05-17T19:32:32.395 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44511.log 2024-05-17T19:32:32.400 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.45780.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45780.log.gz 2024-05-17T19:32:32.401 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37077.log 2024-05-17T19:32:32.402 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.admin.44511.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.44511.log.gz 2024-05-17T19:32:32.404 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50419.log 2024-05-17T19:32:32.404 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.37077.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37077.log.gz 2024-05-17T19:32:32.404 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39103.log 2024-05-17T19:32:32.418 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.50419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-05-17T19:32:32.418 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.39103.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39103.log.gz 2024-05-17T19:32:32.431 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42022.log 2024-05-17T19:32:32.442 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60273.log 2024-05-17T19:32:32.450 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.42022.log: 93.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56909.log 2024-05-17T19:32:32.451 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.60273.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60273.log.gz 2024-05-17T19:32:32.457 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55506.log 2024-05-17T19:32:32.467 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.56909.log: 89.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49637.log 2024-05-17T19:32:32.467 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.55506.log: -- replaced with /var/log/ceph/cluster1-client.mirror.42022.log.gz 2024-05-17T19:32:32.468 INFO:teuthology.orchestra.run.smithi195.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55506.log.gz 2024-05-17T19:32:32.471 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.56909.log.gz 2024-05-17T19:32:32.475 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52550.log 2024-05-17T19:32:32.475 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49637.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49637.log.gz 2024-05-17T19:32:32.484 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48438.log 2024-05-17T19:32:32.497 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.52550.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52550.log.gz 2024-05-17T19:32:32.499 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40309.log 2024-05-17T19:32:32.500 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.48438.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48438.log.gz 2024-05-17T19:32:32.517 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59258.log 2024-05-17T19:32:32.518 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.40309.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40309.log.gz 2024-05-17T19:32:32.520 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50816.log.gz 2024-05-17T19:32:32.521 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62799.log 2024-05-17T19:32:32.521 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.59258.log: 57.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59258.log.gz 2024-05-17T19:32:32.541 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42985.log 2024-05-17T19:32:32.541 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.62799.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62799.log.gz 2024-05-17T19:32:32.550 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46507.log 2024-05-17T19:32:32.560 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.42985.log: 93.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56335.log 2024-05-17T19:32:32.561 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46507.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48695.log.gz 2024-05-17T19:32:32.561 INFO:teuthology.orchestra.run.smithi195.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.46507.log.gz 2024-05-17T19:32:32.562 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53655.log 2024-05-17T19:32:32.564 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.56335.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56335.log.gz 2024-05-17T19:32:32.564 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49814.log 2024-05-17T19:32:32.564 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.53655.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53655.log.gz 2024-05-17T19:32:32.565 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63283.log 2024-05-17T19:32:32.567 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.49814.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49814.log.gz 2024-05-17T19:32:32.568 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58562.log 2024-05-17T19:32:32.568 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.63283.log: 6.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63283.log.gz 2024-05-17T19:32:32.568 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38608.log 2024-05-17T19:32:32.570 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.58562.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58562.log.gz 2024-05-17T19:32:32.570 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42737.log 2024-05-17T19:32:32.570 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.38608.log: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.38608.log.gz 2024-05-17T19:32:32.570 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46926.log 2024-05-17T19:32:32.570 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.42737.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42737.log.gz 2024-05-17T19:32:32.572 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40963.log 2024-05-17T19:32:32.572 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.46926.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46926.log.gz 2024-05-17T19:32:32.573 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47327.log.gz 2024-05-17T19:32:32.573 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41312.log 2024-05-17T19:32:32.574 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.40963.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40963.log.gz 2024-05-17T19:32:32.574 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44309.log 2024-05-17T19:32:32.574 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.41312.log: 73.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41312.log.gz 2024-05-17T19:32:32.583 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53751.log 2024-05-17T19:32:32.585 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.44309.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44309.log.gz 2024-05-17T19:32:32.585 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44189.log 2024-05-17T19:32:32.585 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53751.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53751.log.gz 2024-05-17T19:32:32.586 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36769.log 2024-05-17T19:32:32.588 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.44189.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44189.log.gz 2024-05-17T19:32:32.588 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61553.log 2024-05-17T19:32:32.588 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.36769.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36769.log.gz 2024-05-17T19:32:32.589 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51110.log 2024-05-17T19:32:32.589 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.61553.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61553.log.gz 2024-05-17T19:32:32.591 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33374.log 2024-05-17T19:32:32.591 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.51110.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51110.log.gz 2024-05-17T19:32:32.592 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38195.log 2024-05-17T19:32:32.593 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.admin.33374.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33374.log.gz 2024-05-17T19:32:32.593 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41650.log 2024-05-17T19:32:32.594 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.38195.log: 73.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38195.log.gz 2024-05-17T19:32:32.594 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53913.log 2024-05-17T19:32:32.600 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.41650.log: 96.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.54481.log.gz 2024-05-17T19:32:32.600 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53816.log 2024-05-17T19:32:32.600 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53913.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41650.log.gz 2024-05-17T19:32:32.600 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53510.log 2024-05-17T19:32:32.601 INFO:teuthology.orchestra.run.smithi195.stderr: 94.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53913.log.gz 2024-05-17T19:32:32.602 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53816.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43756.log 2024-05-17T19:32:32.602 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster1-client.mirror.53510.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53510.log.gz 2024-05-17T19:32:32.602 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cluster2-client.mirror.43756.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.53816.log.gz 2024-05-17T19:32:32.603 INFO:teuthology.orchestra.run.smithi195.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43756.log.gz 2024-05-17T19:32:32.647 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49856.log.gz 2024-05-17T19:32:32.717 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50419.log.gz 2024-05-17T19:32:32.908 INFO:teuthology.orchestra.run.smithi195.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42985.log.gz 2024-05-17T19:32:32.926 INFO:teuthology.orchestra.run.smithi134.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-05-17T19:32:33.802 INFO:teuthology.orchestra.run.smithi134.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-05-17T19:32:33.891 INFO:teuthology.orchestra.run.smithi134.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-05-17T19:32:33.893 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-05-17T19:32:33.893 INFO:teuthology.orchestra.run.smithi134.stderr:real 0m6.732s 2024-05-17T19:32:33.894 INFO:teuthology.orchestra.run.smithi134.stderr:user 0m19.686s 2024-05-17T19:32:33.894 INFO:teuthology.orchestra.run.smithi134.stderr:sys 0m0.765s 2024-05-17T19:33:19.476 INFO:teuthology.orchestra.run.smithi195.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-05-17T19:33:21.228 INFO:teuthology.orchestra.run.smithi195.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-05-17T19:33:33.401 INFO:teuthology.orchestra.run.smithi195.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-05-17T19:33:33.403 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-05-17T19:33:33.403 INFO:teuthology.orchestra.run.smithi195.stderr:real 1m6.209s 2024-05-17T19:33:33.404 INFO:teuthology.orchestra.run.smithi195.stderr:user 3m11.269s 2024-05-17T19:33:33.404 INFO:teuthology.orchestra.run.smithi195.stderr:sys 0m7.730s 2024-05-17T19:33:33.404 INFO:tasks.ceph:Archiving logs... 2024-05-17T19:33:33.405 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/log/ceph to /home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219/remote/smithi134/log 2024-05-17T19:33:33.406 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-17T19:33:36.144 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/log/ceph to /home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219/remote/smithi195/log 2024-05-17T19:33:36.145 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-17T19:34:02.733 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json' 2024-05-17T19:34:02.738 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-17T19:34:02.750 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/task/install/__init__.py", line 630, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2024-05-17T19:34:02.756 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-17T19:34:02.756 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-17T19:34:02.770 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-17T19:34:02.863 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-17T19:34:02.874 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-17T19:34:02.875 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-17T19:34:02.877 DEBUG:teuthology.orchestra.run.smithi195:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-17T19:34:02.892 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-05-17T19:34:02.895 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-05-17T19:34:02.896 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-17T19:34:02.897 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-05-17T19:34:02.897 INFO:teuthology.orchestra.run.smithi134.stdout:^* hv01.front.sepia.ceph.com 3 7 377 27 -40us[ -57us] +/- 44ms 2024-05-17T19:34:02.897 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 85 -544us[ -559us] +/- 54ms 2024-05-17T19:34:02.897 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 11 +800us[ +800us] +/- 72ms 2024-05-17T19:34:02.897 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-17T19:34:02.917 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-17T19:34:02.917 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-05-17T19:34:02.917 INFO:teuthology.orchestra.run.smithi195.stdout:^* hv01.front.sepia.ceph.com 3 6 377 23 -31us[ -41us] +/- 44ms 2024-05-17T19:34:02.917 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 88 -564us[ -574us] +/- 54ms 2024-05-17T19:34:02.917 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 +764us[ +764us] +/- 72ms 2024-05-17T19:34:02.918 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-17T19:34:02.918 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-17T19:34:02.929 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-17T19:34:02.930 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-17T19:34:02.941 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-17T19:34:03.008 DEBUG:teuthology.orchestra.run.smithi195:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-17T19:34:03.112 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-17T19:34:03.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-17T19:34:03.143 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-17T19:34:03.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-17T19:34:03.173 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-17T19:34:03.184 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-17T19:34:03.194 INFO:teuthology.task.internal:Duration was 17359.456687 seconds 2024-05-17T19:34:03.195 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-17T19:34:03.252 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-17T19:34:03.252 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-17T19:34:03.255 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-17T19:34:03.293 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-17T19:34:03.312 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-17T19:34:03.640 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-17T19:34:03.640 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2024-05-17T19:34:03.640 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-17T19:34:03.701 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi195.front.sepia.ceph.com 2024-05-17T19:34:03.701 DEBUG:teuthology.orchestra.run.smithi195:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-17T19:34:03.724 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-17T19:34:03.724 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-17T19:34:03.744 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-17T19:34:03.791 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-17T19:34:03.791 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-17T19:34:03.811 DEBUG:teuthology.orchestra.run.smithi195:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-17T19:34:03.953 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-17T19:34:03.965 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-17T19:34:03.965 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-17T19:34:03.990 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-17T19:34:04.015 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-17T19:34:04.027 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-17T19:34:04.033 DEBUG:teuthology.orchestra.run.smithi195:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-17T19:34:04.053 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-05-17T19:34:04.086 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = core 2024-05-17T19:34:04.115 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-17T19:34:04.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-17T19:34:04.134 DEBUG:teuthology.orchestra.run.smithi195:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-17T19:34:04.170 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi195.front.sepia.ceph.com, flagging run as failed 2024-05-17T19:34:04.170 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-17T19:34:04.181 INFO:teuthology.task.internal:Transferring archived files... 2024-05-17T19:34:04.183 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219/remote/smithi134 2024-05-17T19:34:04.183 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-17T19:34:04.215 DEBUG:teuthology.misc:Transferring archived files from smithi195:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219/remote/smithi195 2024-05-17T19:34:04.215 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-17T19:34:46.600 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-05-17T19:34:46.609 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219/remote/smithi195/coredump/1715959799.56225.core.gz: gzip compressed data, was "1715959799.56225.core", last modified: Fri May 17 19:34:04 2024, max compression, original size modulo 2^32 641400832 2024-05-17T19:34:46.609 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2024-05-17T19:34:49.343 INFO:teuthology.task.internal: core looks like: /tmp/tmpoy6300qi: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'rbd-mirror --foreground --cluster cluster2 --id mirror.0', real uid: 1000, effective uid: 1000, real gid: 1316, effective gid: 1316, execfn: '/usr/bin/rbd-mirror', platform: 'x86_64' 2024-05-17T19:34:49.400 INFO:teuthology.task.internal: dump_program: rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-05-17T19:34:49.400 DEBUG:teuthology.orchestra.run.smithi195:> which 'rbd-mirror --foreground --cluster cluster2 --id mirror.0' 2024-05-17T19:34:49.482 INFO:teuthology.orchestra.run.smithi195.stderr:/usr/bin/which: unrecognized option '--foreground' 2024-05-17T19:34:49.483 INFO:teuthology.orchestra.run.smithi195.stderr:/usr/bin/which: unrecognized option '--cluster' 2024-05-17T19:34:49.483 INFO:teuthology.orchestra.run.smithi195.stderr:/usr/bin/which: unrecognized option '--id' 2024-05-17T19:34:49.483 INFO:teuthology.orchestra.run.smithi195.stderr:/usr/bin/which: no cluster2 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2024-05-17T19:34:49.483 INFO:teuthology.orchestra.run.smithi195.stderr:/usr/bin/which: no mirror.0 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2024-05-17T19:34:49.484 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-05-17T19:34:49.484 INFO:teuthology.orchestra.run.smithi195.stdout:/usr/bin/rbd-mirror 2024-05-17T19:34:49.484 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi195 with status 2: "which 'rbd-mirror --foreground --cluster cluster2 --id mirror.0'" 2024-05-17T19:34:49.495 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-17T19:34:49.507 INFO:teuthology.task.internal:Not uploading archives. 2024-05-17T19:34:49.507 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-17T19:34:49.558 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-17T19:34:49.558 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-17T19:34:49.565 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-17T19:34:49.580 INFO:teuthology.orchestra.run.smithi134.stdout: 262165 4 drwxr-xr-x 3 ubuntu ubuntu 4096 May 17 19:34 /home/ubuntu/cephtest 2024-05-17T19:34:49.581 INFO:teuthology.orchestra.run.smithi134.stdout: 262166 4 drwxr-xr-x 4 ubuntu ubuntu 4096 May 17 19:34 /home/ubuntu/cephtest/archive 2024-05-17T19:34:49.581 INFO:teuthology.orchestra.run.smithi134.stdout: 403403 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 17 19:34 /home/ubuntu/cephtest/archive/audit 2024-05-17T19:34:49.581 INFO:teuthology.orchestra.run.smithi134.stdout: 403703 120 -rw------- 1 ubuntu ubuntu 119528 May 17 19:34 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-05-17T19:34:49.581 INFO:teuthology.orchestra.run.smithi134.stdout: 262168 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 17 19:34 /home/ubuntu/cephtest/archive/syslog 2024-05-17T19:34:49.581 INFO:teuthology.orchestra.run.smithi134.stdout: 262209 4 -rw-rw-rw- 1 ubuntu ubuntu 29 May 17 14:44 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-05-17T19:34:49.581 INFO:teuthology.orchestra.run.smithi134.stdout: 265295 184 -rw-r--r-- 1 ubuntu ubuntu 187430 May 17 19:34 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-17T19:34:49.581 INFO:teuthology.orchestra.run.smithi134.stdout: 265296 4 -rw-rw-rw- 1 ubuntu ubuntu 29 May 17 14:44 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-05-17T19:34:49.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-17T19:34:49.582 INFO:teuthology.orchestra.run.smithi134.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-05-17T19:34:49.583 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ee59d86aad04089481e55c1a15aa5871339935b4/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cad7edd8d899a7c59a48f7da3294fca4cb778f64/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-05-17T19:34:49.591 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-17T19:34:49.603 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-17T19:34:49.616 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: error during thrashing 2024-05-17T19:34:49.616 INFO:teuthology.nuke:Checking targets against current locks 2024-05-17T19:34:49.622 INFO:teuthology.orchestra.run.smithi195.stdout: 262165 4 drwxr-xr-x 4 ubuntu ubuntu 4096 May 17 19:34 /home/ubuntu/cephtest 2024-05-17T19:34:49.623 INFO:teuthology.orchestra.run.smithi195.stdout: 262166 4 drwxr-xr-x 5 ubuntu ubuntu 4096 May 17 19:34 /home/ubuntu/cephtest/archive 2024-05-17T19:34:49.623 INFO:teuthology.orchestra.run.smithi195.stdout: 403403 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 17 19:34 /home/ubuntu/cephtest/archive/audit 2024-05-17T19:34:49.623 INFO:teuthology.orchestra.run.smithi195.stdout: 403407 124 -rw------- 1 ubuntu ubuntu 123732 May 17 19:34 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-05-17T19:34:49.623 INFO:teuthology.orchestra.run.smithi195.stdout: 262168 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 17 19:34 /home/ubuntu/cephtest/archive/syslog 2024-05-17T19:34:49.623 INFO:teuthology.orchestra.run.smithi195.stdout: 262209 4 -rw-rw-rw- 1 ubuntu ubuntu 29 May 17 14:44 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-05-17T19:34:49.623 INFO:teuthology.orchestra.run.smithi195.stdout: 265295 184 -rw-r--r-- 1 ubuntu ubuntu 188091 May 17 19:34 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-17T19:34:49.623 INFO:teuthology.orchestra.run.smithi195.stdout: 265296 4 -rw-rw-rw- 1 ubuntu ubuntu 29 May 17 14:44 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-05-17T19:34:49.623 INFO:teuthology.orchestra.run.smithi195.stdout: 262167 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 17 15:29 /home/ubuntu/cephtest/archive/coredump 2024-05-17T19:34:49.624 INFO:teuthology.orchestra.run.smithi195.stdout: 266683 53868 -rw------- 1 ubuntu ubuntu 641400832 May 17 15:29 /home/ubuntu/cephtest/archive/coredump/1715959799.56225.core 2024-05-17T19:34:49.624 INFO:teuthology.orchestra.run.smithi195.stdout: 266682 67112 -rw------- 1 ubuntu ubuntu 744906752 May 17 15:29 /home/ubuntu/cephtest/archive/coredump/1715959788.52002.core 2024-05-17T19:34:49.624 INFO:teuthology.orchestra.run.smithi195.stdout: 403405 4 drwxr-xr-x 3 ubuntu ubuntu 4096 May 17 14:52 /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-05-17T19:34:49.624 INFO:teuthology.orchestra.run.smithi195.stdout: 403406 4 drwxr-xr-x 3 ubuntu ubuntu 4096 May 17 14:53 /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-05-17T19:34:49.624 INFO:teuthology.orchestra.run.smithi195.stdout: 417847 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 17 14:53 /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-05-17T19:34:49.625 INFO:teuthology.orchestra.run.smithi195.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-05-17T19:34:49.646 DEBUG:teuthology.nuke:shortname: smithi134 2024-05-17T19:34:49.646 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-17T19:34:49.668 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-17 14:35:49.117998', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHYe1ZGnxZjX7+sQweRvnThF1q47PodBSwam7dcyTHgN8O9FeALEpBU/YCmwo/S2di3fktueeGSo22ojmFO2hnY='} 2024-05-17T19:34:49.671 DEBUG:teuthology.nuke:shortname: smithi195 2024-05-17T19:34:49.672 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-17T19:34:49.697 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-05-17_14:29:27-rbd-wip-dis-testing-distro-default-smithi/7711219', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-17 14:35:49.119830', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHOwvwP+xWhFfbARMc1jjL85g4dIJywrcGaxZwuxpRdYo+R7TPpbE+bquiE8iV2Wra/zwrM6BG46opM+s86KYvw='} 2024-05-17T19:34:49.723 INFO:teuthology.orchestra.console.smithi134:Power off 2024-05-17T19:34:49.723 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-17T19:34:49.750 DEBUG:teuthology.orchestra.console.smithi134:power off output: Chassis Power Control: Down/Off 2024-05-17T19:34:49.750 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-17T19:34:49.763 INFO:teuthology.orchestra.console.smithi195:Power off 2024-05-17T19:34:49.764 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-17T19:34:49.776 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-05-17T19:34:49.787 DEBUG:teuthology.orchestra.console.smithi195:power off output: Chassis Power Control: Down/Off 2024-05-17T19:34:49.787 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-17T19:34:49.810 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-05-17T19:34:53.777 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-17T19:34:53.811 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-17T19:34:53.889 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-05-17T19:34:53.921 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-05-17T19:34:57.894 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-17T19:34:57.923 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-17T19:34:58.006 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is off 2024-05-17T19:34:58.033 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off 2024-05-17T19:34:58.107 INFO:teuthology.orchestra.console.smithi134:Power off completed 2024-05-17T19:34:58.134 INFO:teuthology.orchestra.console.smithi195:Power off completed 2024-05-17T19:34:58.240 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} duration: 17359.456686735153 failure_reason: 'Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi195 with status 110: ''mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ee59d86aad04089481e55c1a15aa5871339935b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\''\'' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh''' flavor: default owner: scheduled_dis@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e7d6cbdcf9eb45e18430628889b696e6 status: fail success: false 2024-05-17T19:34:58.241 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-17T19:34:58.328 INFO:teuthology.run:FAIL